How to gather usage statistics for various open source testing frameworks
My organization would like to choose between the TestNG and Robot Framework. I thought it better to approach this from the stand point of usage statistics, since both are open source. How do I go about doing this. I did not find any head to head comparison information when I looked it up and so I thought it better to go with usage stats. Could someone point me in the right direction?
By the way we're a product company providing solutions for the Supply chain Management domain. We have both J2EE Web applications (flagship) and mobile applications (iOS and Android)
Currently we're white-boxing with Groovy/testNG but there's confusion for black boxing. We're planning to use Selenium as the automation tool with either TestNG or Robot.
I think you want to pick the framework which has the most active development and the most documentation resources on the internet.
Of-course you first need to check which framework fits your requirements, I would pilot all for a short while (starting with the most active one. If you have multiple candidates.)
Which is more popular:
- Number of (recent) commits in the version control system
- Number of contributors in the version control system
- Number of tags on StackOverflow (Robot + RobotFramework:500 vs TestNG:1665)
- Number of Google results
- Active users on the IRC channel on irc.freenode.org
In the past we have picked a slightly "worse" framework, because it was actively developed compared with the single developer for the other framework.
Other things I would look into are:
- Is the framework in a programming language your team has experience with
- Does the framework have good test coverage
This will make it easier to extend it in the future. When the community does not fix a defect your are running into.