What issues, when automating responsive mobile sites, don't appear on a desktop
We're trying to have automated mobile testing but it's taking a huge amount of effort and I'm not seeing any rewards.
Our site is responsive and has a mobile view. We do not have native mobile apps.
What I have found so far is that nearly all the errors we have found also occur on desktop flows, for example
A required field wasn't entered
A field entry used an invalid format
I got a 404 or 500 error page
A dropdown list did not have the expected values
An error field did not have the correct error highlighting
I have not been able to identify any mobile specific issues (issues that would not occur on desktop) that our automation would catch.
Also, the few errors that I have seen such as :
A field is on the page but not currently visible in the viewport
A field is to small to tap on with a finger
Tapping a field twice leads to a click on the next page when finally loaded
are NOT the sort of issues that our selenium based automated mobile testing (using Ruby, Capybara and Appium) would capture.
I am being asked by my organization to do automated mobile testing because this is half of our traffic. I accept this but feel that given the difficulty of setting up mobile testing and the effort involved, I should identify what issues automated mobile testing should be expected to uncover.
I am looking for issues that wouldn't be present when running desktop automation on a narrow window to force the responsive mobile view.
This follows a theme that I have been focused on a lot lately, which is 'How to make sure tests add value'
Agree. Faced same condition before. As experienced earlier few things might help you at some extend.
You need to convey them the pros & cons of the automation testing.
You need to give them a brief about the use of automation as well. I mean with lesser knowledge one can expect to much from you. You need to take initiatives to keep balance between actual possible things to automate and expectations
As per my knowledge, In your case of browser automation we can ensure that everything is working well 24*7. This is a primary goal of browser automation.
Communicate clearly if we are not bothering about things should work ALWAYS then we should think of reduce the dependency & efforts on browser automation
Limitations: Its really out of scope [as of now at lease] to automate browser to catch images or content overlapping, veriication of image quality and visibility, crosschecking on different versions of browser in [responsive] website