Experiences with automation of hybrid mobile apps
I am looking into the UI test automation options for hybrid apps (Cordova PhoneGap) on both iOS and Android. So far I still struggle with finding any decent tool. I may consider proprietary product (TestDroid, SilkMobile, Ranorex or so..), but first I would really like evaluate available open-source toolset. Yeah, there are many mobile testing frameworks already such as UIAutomator, Robotium, Espresso for Android, UIAutomation or KIF on iOS. Problem is those tools are closely bound to specific platform. So I favor idea of multiplatform tools such as Calabash, Appium, MonkeyTalk. The latter two seems to be most popular and feature-rich alternatives. I tried them both, however, I found none of them mature enough, each of them having a lot of "infancy" issues. Appium's concept is really nice but even now, with version 1.0, it is being struck down by poor documentation, outdated examples and some hybrid app issues (selendroid dependancy - there were reported issues with coordination between Appium and Selendroid API). It is simply painful to setup locally (and I cannot only depend on Sauce Labs as one needs to debug the tests locally). MonkeyTalk seems more user friendly and better documented, though I experienced some issues as well (crashes, instrumentation failures) + some of the better features are not available for iOS (on-the-fly instrumentation..). The better equiped Pro version is still Beta after all. It seems there is no ultimate solution for hybrid apps automation as of yet. The whole field of mobile testing looks still, surprisingly, undeveloped. While I favor the idea of having one tool for all mobile test automation, the most critical requirement is still "to have a tool that offers best test automation capabilities for certain app". If this means I need to use two or even three different tools best suiting specific mobile platforms, so be it. But I cannot be the only one here, trying to solve this dilemma, so I wonder what has your experience been in building the best possible infrastructure for testing mobile hybrid apps? Please no self-promotion of test automation tools vendors.
I have used eggPlant by TestPlant with success. Pros: Can be used on pretty much any platform that you can remote into Has a very good image compare solution with good flexibility/resiliency I have used this with great success on iOS, Android, Linux, Windows, Mac Cons: Uses image compare Uses a proprietary language Expensive I have also used Appium, although it is, as you say, a bit underdeveloped. However, the shop I currently work for are making good progress with Appium now on hybrid apps with iOS and Android. Pros: Free (as free as an economist would allow) Well-supported Nice coordination with Selenium Cons: New Needs a bit more technical skill to get up and running Also of note: SOASTA and Perfecto Mobile As a note, I have worked at many high-performing companies, and have successfully implemented and supported automation efforts.