How to approach running automated tests "from the head"?
I'm currently one of a group of developers developing tests for a web-based line of products. These products are separate but related, and are periodically released (ie versioned).
We are trying to make our tests all "run from the head" against all different versions, meaning that the current state of our master test code branch can run meaningfully against arbitrary versions of the production builds. At first, this seems a bit intimidating, particularly since our tests are generally UI (Selenium) based, and hence, quite dependent on particular versions.
Any tips for making this process easier? Or at least, more doable?
You are astute to recognize that change is the Achilles Heel of automated UI tests. This is no different from developing/maintaining a shared library that needs to work with multiple versions of an application.
There are practices you can follow to reduce (but not eliminate) the pain. On the product side:
- Use predictable element IDs in your UI so that tests are less reliant on the DOM layout.
- Design your pages using reusable components so that the page design is more consistent; this reduces the number of patterns the test needs to understand.
- Get your UI developers involved in the test design so that everyone understands the impact of changes.
On the test side:
- Take advantage of element IDs when they are available (and when they are consistent; dynamic element IDs are not particularly useful to a tester).
- Structure your tests to take advantage of page design patterns.
- Allow your tests to be aware of what product version they are running against, and tag your tests with version information.
- Consider negotiating the number of production versions you intend to support. Perhaps there is some middle ground between compatibility with only the latest code and compatibility with arbitrary versions.
- Consider how much of test suite needs to remain compatible. Some tests may be more crucial than others; it may be enough to commit to compatibility for only a subset of the full test suite.