How to manage application data when running multiple tests in parallel?
Our functional test suite is slow
I have inherited a functional test suite using Selenium 2 (Webdriver) and Junit. We have around 500 tests written for our Java web application. Running the full suite takes several hours, so we only run it overnight on our CI server. I want to speed up our test suite by running our tests in parallel, I have researched various solutions for parallel execution including using Selenium Grid.
How we currently handle data
Our test suite currently drops and restores the database in between each test class, we do this to have a known state and avoid data conflicts between tests. However, if we were to run several tests in parallel restoring the database would likely results in errors unless it was coordinated across tests (I wouldn't want to drop the database while another test is in the middle of its execution).
Some possible solutions:
Here are some ideas:
- Create test groups (ex: tests which can run concurrently vs tests that have to run sequentially)
- Have each test remove the data it creates (not always possible)
- Run each tests using a different user to minimize data conflict (very time consuming to create 500 users)
- Deploy multiple instances of our web application, each with its own database (each test would have to known which instance its running on and drop the correct database)
Edit: These guys touch on this issue http://youtu.be/oX-0Mt5zju0?t=9m10s
Your nice list covers all of the usual bases.
- Implement some sort of "critical region" mechanism around the individual pieces of data that are shared among tests, and that different tests require to have different values.
This creates its own problems, of course. It causes one test to block while waiting for another to finish with the data. If the data is held long enough by a test, other tests might time out while waiting (if you have a timeout on your tests, or if you are running on some service like Sauce Labs that might abort an idle browser session).
So use this trick only if the data conflicts are few and short-lived.
The long runtimes might also benefit from another idea: Use the UI for only those parts of the test that require the UI. Skip around the UI to some faster mechanism for setups and teardowns, and perhaps for assertions (if you're not asserting something about how information is displayed through the UI).