What Test Automation Product delivers management, execution and results?
I am looking into ways of managing my automated tests through some kind of framework or management tool. While there is a lot of software that does the grunt work, such as Selenium, CodedUI etc, I am interested in views on tools to actually manage the tests, execute and report results back.
Doing some Google'ing I have come across the following - can anyone give me any views from their experience? Are there any other recommendations?
- AXE - http://www.axetest.com
- Robot Framework - code.google.com/p/robotframework/
- Cirro Framework - http://www.cirrotest.com
- Ranorex - www.ranorex.com
- QA Complete - www.testcomplete.com
Any advice appreciated.
As you've noted, there are any number of these frameworks. There isn't really all that much difference between them in terms of reporting results - that tends to be a matter of preference more than anything else.
Where the big difference lies in is execution handling. What you want to look for is something that integrates easily with your existing automation. For instance, if you're working with CodedUI, Microsoft's Team Foundation Server and Microsoft Test Manager/Lab Manager work together seamlessly once TFS has been configured for lab management (the module that handles automation execution). QA Complete integrates seamlessly with TestComplete, and so forth - all the big box solutions include easy integration with their proprietary automation tools.
The lower-cost (and free) tools usually operate through an API that you'd need to program against. Some don't handle execution on their own, but offer an API that allows test results to be posted to them for reporting purposes (TestLink is one free tool I know of that does this).
Apart from that, I'd look for something that meets your budget and is reasonably easy to work with. Chances are unless you're using a big-box integrated system like Microsoft TFS/MTM/TLM/CodedUI, SmartBear QAComplete/TestComplete, HP Quality Center or similar, you'll have to do some work to build the integration so a clear API is a good thing.