What test management system should I use?
There must be many questions like this on the site but I have some unique requirements that affect my choice of test management / QA process management system. This is for managing the test cases and recording results, not for performing the actual tests, which will be done manually and with other automation tools.
We're building tests for a website that has at least a few changes in behaviour every day. So if there are 1,000 tests, at least 5 will definitely break every day because of a new feature or design change. I mention it because I don't want to use something that assumes once a test case is defined it will never change or changing it requires a formal review process.
I want to write a test case 1 time, then record many variations of the same result. One for each device type, browser, OS, variations in other. For example, the "User Sign Up" main test case is defined once, then the result recorded every day for each browser/device/OS combo.
I'd like it to be web-based. Also with some kind of API to let our automated test tools (which mainly use Selenium) to post-test results as they happen).
There are a number of systems that can handle this. I'm familiar with Test Link and TFS, both of which have APIs (although the TFS API is a little challenging - speaking as someone who's worked out how to post test results to TFS through the API) and both of which have ways to handle test cases that need to change.
A lot depends on your budget - Test Link is free (as in open source, and as in beer), TFS is not (Microsoft Team Foundation Server if you're not familiar with the abbreviation). QA Complete from SmartBear is another commercial tool that offers the functionality you're looking for.
Here's how these tools (and I suspect the other major commercial offerings) work:
You define one or more test environments - this covers your device types, browsers, OSs and so forth. Depending on the tool you define the environments required at the test suite, test plan, or test case level. The API to communicate the tool varies but can be worked with. For TFS it needs a fair amount of code and assorted other dependencies taken care of before you start the tests. Test Link also has dependencies that need to be set up in the tool before you start.
For test case updates, the usual arrangement is that the test case is versioned once it has been run (that is, there's a result posted to the tool). The rationale is that if the test case has to change, the historical results will be invalid unless you retain the history of the test case itself. Updating changed test cases is usually a matter of creating a new version (and incrementing the test case versioning), then running against that until there's another change.
Depending on the tool, you may be able to break down your test cases into reusable steps that minimize the amount of re-versioning you need to do.
I know Team Foundation Server allows this through what it calls "Shared Steps" - which are pretty much what they sound - one or more steps in a test case which can be used by multiple test cases. (Caveat - Team Foundation Server needs to use Microsoft Test Manager for having test suites and test plans defined - currently, it's not possible to create these through the TFS interface - and MTM can't be used without TFS).
I'll often set up a single step as a shared step so it can be used by any test case (such as "click the login button" which doesn't care what the user credentials are). You can also parameterize steps so that instead of a whole lot of steps for each different value your tests will enter into a particular field, you have one step to enter "@value" into the field, with the values that can be used defined by the test case or an attached data file.
Test Link doesn't have that level of reusability, that I know of. I don't know enough about other tools to offer any suggestions.
There is a good listing of test management tools available in the Test Management section of the Ministry of Testing's test tool listing. (It's a long list... you'll need to do a fair amount of research in this)
On the plus side, most of the commercial tools do offer some form of trial. I know Microsoft has a free service for TFS that they host (Team Foundation Service) and contains the most up-to-date feature set. That's free for up to 5 users per account - I don't know any other details.
Good luck finding something that meets your needs.