Automated Test Management: Running, Saving Results, Querying, etc
The team I am currently working with has a very large application to test (several custom programs, running in a distributed environment), and has built up a very large set of automated test cases for regression and feature testing. These tests are large and there are a lot, so full test runs are dispatched across many machines, the results gathered, and then imported into a custom web app. There are some very nice tools out there for writing the automated tests, but there doesn't seem to be much for managing a "automated testing workplace". We ended up writing custom applications to do all of the following: Schedule tests to run on one of several machines Gather results, import into database Web application for result viewing and querying Results per run/software version/test machine/etc Integration with external bug tracker for tracing failures to tests Nice to have: Smart analysis of results "This test started failing/passing" "This test is unreliable/intermittent" etc Does anyone know of anything off the shelf that handles all of the above? Most of what I have found before is very focused on manual testing, which we don't do much of.
If I'm understanding correctly you could use a Continuous Integration tool like: Jenkins Jenkins is an award-winning application that monitors executions of repeated jobs, such as building a software project or jobs run by cron. Team City TeamCity is a user-friendly continuous integration (CI) server for professional developers and build engineers, like ourselves. It is trivial to setup, manage and use. It scales perfectly for a company or a team of any size. Both have numerous features, which you can browse via their respective websites. We are currently using Jenkins to run our automated tests (smoke, regression and unit tests). It does a nice job of reporting and is customizable so that you can a variety of plugins (which are available via the Jenkins site). We push tests to different machines and test results for the tests are reported within Jenkins.