How to gather usage statistics for various open source testing frameworks



  • My organization would like to choose between the TestNG and Robot Framework. I thought it better to approach this from the stand point of usage statistics, since both are open source. How do I go about doing this. I did not find any head to head comparison information when I looked it up and so I thought it better to go with usage stats. Could someone point me in the right direction?

    By the way we're a product company providing solutions for the Supply chain Management domain. We have both J2EE Web applications (flagship) and mobile applications (iOS and Android)

    Currently we're white-boxing with Groovy/testNG but there's confusion for black boxing. We're planning to use Selenium as the automation tool with either TestNG or Robot.



  • I think you want to pick the framework which has the most active development and the most documentation resources on the internet.

    Of-course you first need to check which framework fits your requirements, I would pilot all for a short while (starting with the most active one. If you have multiple candidates.)

    Which is more popular:

    • Number of (recent) commits in the version control system
    • Number of contributors in the version control system
    • Number of tags on StackOverflow (Robot + RobotFramework:500 vs TestNG:1665)
    • Number of Google results
    • Active users on the IRC channel on irc.freenode.org

    In the past we have picked a slightly "worse" framework, because it was actively developed compared with the single developer for the other framework.

    Other things I would look into are:

    • Is the framework in a programming language your team has experience with
    • Does the framework have good test coverage

    This will make it easier to extend it in the future. When the community does not fix a defect your are running into.



Suggested Topics

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2