Screen Capture Website images for UI regression testing
I have been playing with selenuim & MS CodedUI tests but these all seem very complicated and more suited to a developer.
Is there a simpler tool that will allow me to take screen shots of sections of a page, and if that section changes, the test fails?
Is my idea flawed or incorrect?
Should I be asking how should I create robust and non fragile Website UI regression tests and what tools will help non developers create this.
The long term goal in the future one day, will be to create a robust and automated build service that can verify that the site is in a releasable state.
For really good, robust, maintainable UI automated regression, you need code. Using screenshots is bad for several reasons:
- Even with a fuzz tolerance built in to prevent things like the exact color of the display throwing false positives, there are too many trivial things that can throw a false positive.
- A change of display resolution will utterly destroy your regression.
- You still have to interact with the application to get to the screens you're wanting to check. That's going to take either a lot of repetitive record/playback (bad) or well-optimized coding, particularly if you're dealing with the kind of application that doesn't do anything unless you're logged in with the proper credentials.
- Screenshots don't tell you if the data your application is storing is correct.
- For look and feel, there is no better tool than the human eyeball.
- A human will always be better at determining whether a change to a screen is significant or not. A computer comparison algorithm will tell you that something has changed. It won't tell you if the change matters.
- The UI of your web application is going to be different depending on the browser running the tests. Rendering of form fields varies dramatically between browsers. If you maintain a set of screenshots for one browser, you won't know if something has caused a rendering issue on a different browser. If you commit to automated testing against multiple browsers, you're talking about maintaining a massive amount of frequently changing screenshots. It's a maintenance nightmare.
Properly done automated functional regression is code.
That said, Selenium and CodedUI are probably not the best tools for someone with limited coding experience. The tools I know that could meet your needs as stated are:
- SmartBear's TestComplete has what they call "keyword testing" which does most of the heavy lifting for you in terms of code. The licensing model is one of the cheaper ones for big box tools. There are a number of free to view webinars on the SmartBear site that demonstrate how the tool works (I am not affiliated with SmartBear in any way, but I used TestComplete for nearly 7 years).
- TestAnywhere is an image-based tool that's reasonably sophisticated and can probably do what you want without overwhelming you. I personally found it too limited for what I needed, but then I'm comfortable with coding and have worked as a programmer in the past. It uses a hybrid of object identification and screenshots of individual fields to identify items.