How to manage a lot of automated test cases?
We are a team of 8 QA Analysts. We have automated more than 3,000 test cases for our application. As this number grows bigger and bigger, we are having issues controlling tests that are already created and I think we have a lot of duplicated test cases. Not only duplicate, but also tests that are already included in another test (Test X does what Test Y already does plus other things).
I was wondering how you guys manage your test cases do avoid having duplicated or not useful ones.
We are using TFS and Microsoft Test Manager and our tests are automated using Selenium.
This problem hits everyone with a decent-sized application sooner or later.
Some of the things you can do to help manage your test cases more cleanly:
- Use a self-documenting system - if you're coding Selenium with C#, use the MS XML commenting and something like Sandcastle to publish the documentation that's generated on build to a common site. That way everyone in the team can search for test cases that do what they need before creating a new one.
- Try to string tests together in a sequence rather than include extras in another test - what I mean here is instead of having test A log on and perform action X, you have test A log on, test B performs action X (with a prerequisite of test A having been completed successfully) and so forth. This does introduce extra dependencies into your test runs but it keeps your test code base more DRY.
- Create library routines for common functions - this includes library routines for logging on, navigating through the application, and so forth - essentially anything that's stable in the application and/or used a becomes a library and you focus the actual tests on the aspects of the application that need more intense examination.
- Review and refactor - your automation code needs to be reviewed and refactored at least as much as production code, and regular reviews/refactoring will help to locate and clean out duplicates and unnecessary tests.
None of these are going to solve your problem. You're always going to have the tension between keeping your tests self-contained and keeping your maintenance load as low as possible. These are just some suggestions to help you identify where you want the balance between lower maintenance (DRY code) and self-contained tests.