Selecting the Right Mobile Test Automation Strategy

• Cognizant 20-20 Insights

Selecting the Right Mobile Test Automation Strategy: Challenges and Principles Understanding the various options and choosing the tools and automation processes best suited to testing both mobile Web and native mobile apps can reduce complexity and speed time to value.

Executive Summary through test automation. As with conven- tional applications, there is an increased need The mobile phone has transitioned from its tra- to automate mobile applications testing to ditional role as a mere communication medium improve scale and efficiency. A well-orchestrated to that of a multipurpose personal gadget. Tech- automation strategy enables hardware manufac- nological advancements and the proliferation turers and software developers to reduce effort of devices across operating systems (Apple iOS, and accelerate time to market. Android and Windows Mobile) and platforms have made it more challenging for hardware manufac- Mobile Technology’s Growth Promise turers and application developers to develop and roll out new products. The growth of mobile devices is evidenced by the following: For vendor peace of mind, mobile applica- • Globally, the user base of location-based tions must be tested to ensure they run on key services has grown from 96 million in 2009 platforms and across a multitude of networks. to more than 526 million in 2012, according to Despite the pressures of short mobile develop- Gartner Research.1 ment cycles, quality testing of applications across operating systems, device platforms and networks • Worldwide mobile advertising and content is a necessary but daunting task to ensure revenue is forecast to reach $67 billion this 2 long-term success in what is a highly fragmented year. and competitive global market. Moreover, non- • IDC predicts that smartphone shipments will functional testing — including usability, security reach 978 million in 2014.3 and adaptability — is as critical as functional By 2016, smartphones and tablets will testing. Effective testing can help device makers • put power in the hands of a billion global and application developers collect appropriate consumers, according to Forrester metrics that improve product quality. Research, Inc.4 This white paper explores industry best practices • Mobile device makers are providing on mobile testing and demonstrates effective customized platforms to increase their share ways of managing mobile application quality of the customer’s wallet.

cognizant 20-20 insights | september 2012 Two Key Challenges in Mobile consider a test case that is created and needs to Application Test Automation be executed on three different platforms — iOS, Android and Blackberry. In this example, the test Mobile Platforms and Technology Coverage case is expected to be executed on 13 different The primary factor that determines an automation devices and in 10 different combinations, which tool’s success is its ability to work across platforms leads to 130 rounds of testing. Designing a test and technology stacks. The following challenges execution matrix of this scope is both difficult and influence automation success: time-consuming (see Figures 1 and 2).

• Device Diversity: Mobile Testing Industry Opportunities >> Multiple platforms and browsers. In the growing mobile market, the need for spe- >> Rendering differences. cialized mobile testing is growing. The following >> Mobile devices with varied application run three opportunities contribute almost 80% of times. total market demand.

• Network Challenges: Mobile Automation Testing >> Multiple network types (e.g., GSM/GPRS/ There are only a handful of tools available to suc- Wi-Fi/Wi-Max). cessfully automate testing. The ROI promised by >> Different speeds of connectivity across automation testing highlights the need for more geographies. options.

>> Multiple network operators with custom- Functional Testing ized network features. Mobile functional testing always involves a non- • Hardware Challenges: heuristic component that cannot be automated. Limitations in processing speed. >> Mobile Performance Testing >> Limitations of mobile memory size. Compared with a desktop/notebook environment, >> Differences in device communication mobile device system resources (e.g., processing protocols (e.g., WAP/HTTP). power, memory, etc.) are limited. Hence, performance testing of mobile applications is crucial. Test Execution Challenge Therefore, this domain is growing and has great Test execution across different devices and brows- potential. ers is a huge, challenging effort. For example,

Real-time Scenario for a Smartphone-based App: Sample Test Execution

Mobile Applications

Use Case 1 iPhone

Use Case 2 Wi-Fi Blackberry

Use Case n Mobile Test Lab Android Emulators Real Device Device Anywhere

iOS Native Android Native Blackberry Native Opera Devices/Browsers Opera Mini Sky Fire Firefox 4 Firefox 5 Firefox 6 Dolphin 3,4&5 (2.1,2.2,2.3, 3.0,4.0) 5,6 ,7,8 & 9 Mobile 11.1

iphone3 iphone4 iphone5 Nexus One Moto Droid Samsung Captivate HTC Incredible Nexus S Moto Droid X Curve 8330 Tour 9630 Bold 9650 Bold 9000 Touch

Figure 1

cognizant 20-20 insights 2 Smartphone Platforms and Browser Combinations

iOS Android Windows Phone Blackberry OS Symbian

Current Version 6.0 Current Version 4.2 Current Version 7.5 Current Version 7.0 Symbian 4 Key Versions Key Version Key Version 2.3, 3.0, 4.0 Key Version 6.5 (iPhone 4.0, iPad 2) 6.x & 7.x series Open Source Closed Source Open Source Closed Source Closed Source Symbian Apple Google Microsoft RIM Multitasking Multitasking Multitasking Multitasking Multitasking

Safari Web Browser Chrome/Webkit Browser Internet Explorer Webkit Browser Webkit Browser

Figure 2

Mobile Automation: What and Why? • User agent-based automation: This approach leverages user agent add-ons that come with Mobile applications are becoming increasingly popular browsers such as Mozilla Firefox sophisticated, significantly increasing the require- and Google Chrome. The user agents help ment for functional testing (see Figures 3 and render the specific Web content that would be 4). To tackle this, test organizations are increas- displayed on the device onto a regular desktop ingly exploring alternatives to traditional manual browser. The functionality can be automated testing. Automated testing is a highly effective by using popular tools such as QTP, Selenium approach to mobile app QA that can provide or RFT — each of which supports all desktop significant business returns, provided it is imple- browsers. mented by using the right tools and architecture, factoring in cross-platform challenges. • Cloud-based automation: Cloud-based test automation vendors support Web-based QA The following specific types of testing also need across platforms. Cloud-based automated to be automated: solutions are highly effective as they are offered as a unified package. • Cross-platform compatibility testing: This is necessitated by the growing number of • Lab-based automation (devices/simulators): handsets and platforms. This method involves setting up tools in a test environment and leveraging simulators User experience testing: Most testing orga- • or physical devices to automate testing using nizations have limited experience with design popular open-source tools such as Selenium. and execution of usability tests. Web driver–supported device browsers and • Field or network testing: Testing must be simulators are best suited here. performed in a geographically distributed environment to account for a variety of network Approaches for Native and Hybrid types. Mobile Applications • Structural challenges: Integrating mobile Cloud-based Approach testing toolsets into existing IT systems is a Cloud-based test automation is one popular crucial hurdle. means for achieving native and hybrid types of Test Automation Mechanisms test automation. The method is useful when for Mobile Apps long-term automation is required across a set of predefined devices. Commercial tool add-on A mobile Web site can be automated in one of the mechanisms such as HP QTP and IBM RFT are also following three manners: available for test engineers who are familiar with

cognizant 20-20 insights 3 Three-tier Mobile Application Testing Framework

Backend Middleware Device UI

Admin Services Enterprise Mobile Client Data Communication Mobile Web Framework/ Mobile Website Application Services Server Application Layers Data

Integration Testing SOA Testing Device Management Installation Functional Testing Testing Testing Integration Testing Security Testing Compatibility Device Testing Proﬁling

Types of Testing Authentication Testing UI Testing Interruption

Fully Automatable Partially Automatable Not Automatable

Figure 3 industry-wide products. The ability to automate in enabling automation in a closed environment, from anywhere is a great advantage: All that for continuous automation or to achieve test is needed is connectivity to the cloud lab. The regression functionality. They also cover nonfunc- “automation-from-anywhere” feature is a distinct tional scenarios that handle mechanisms largely advantage when using tools such as Perfecto relevant to mobile apps. Mobile and Device Anywhere. Open Source/Platform-specific Tools Lab-based Approach Device platform-specific tools reliability is a high There are multiple tool sets with different capa- priority. Platform vendors supply tools such as bilities available for organizations interested in Robotium for Android and iOS instruments for a lab-based approach. Tools such as EggPlant, iOS to provide this facility. However, to leverage Jamo, See Test, ZAP, SilkTest and Test Quest this effectively, it requires additional program- provide industry tool add-ons. They are useful ming skills.

Potential Areas of Automation in End-to-End Mobile Application Testing

100

40 using automated tools 30 Percentage of tests performed

0 Mobile Functional Mobile Mobile Cross- Security Testing SOA/ Package Enterprise Automation Testing Performance Browser Middleware Application Application Testing Testing Testing Testing Testing (e.g., ERP) Testing

Source: Cognizant; based on 350-plus projects performed across all vertical markets and our analysis of industry trends. Figure 4

cognizant 20-20 insights 4 Cognizant’s Tools journey: We have our own platform to easily automate • Perform tool feasibility tests, because mobile testing of mobile native and Web apps automation technologies and platforms vary. across platforms. • Select tools that support both platform simulators and devices, because you can mix Our platform ensures single type scripting that and match devices and simulators to optimize works cross-platform and thus guarantees greater runs on different platforms. reusability of scripts. The platform also offers an easy interface for testers to auto-generate • Aim for automation in nonfunctional areas scripts. For instance, scripts can be automated — interruption and hardware scenarios that without any programming or scripting language include battery state changes. knowledge. It also comes with a facility to • Always optimize on platform support: In some automate as applications are created to support cases, there may be a need for one or more the Agile development methodology. tools to perform automation. Mobile Application Testing Tools • Identify multiple device support and versions Selection Criteria support. There are many successful techniques that can • Determine the use of utilities and reusable make testing applications on mobile devices functions that add value to automation. more effective and efficient. It is essential to • Ensure ROI per each mobile platform and ensure functional and nonfunctional compliance duration of execution as part of the tool con- of mobile applications on devices and mobile sideration process. automation. It is also key to establish labs for testing mobile applications. • Look for avenues for early automation, because most mobile projects are Agile. Some basic criteria for mobile application testing • Integrated execution with the test manage- include the use of emulators, multiple browsers, ment tool and application lifecycle manage- different operating systems and the unique types ment is an important success factor for a tool. of testing required for GUI, field and interrup- Look for data-driven automation support: tion testing. There are roughly 10 key principles • Iterations in the execution will increase that need to be considered while selecting the coverage and ROI. Good candidates for right tool. automation are test cases that have repeated 10 Key Principles in Selecting the Right Tools business logic, functional flows or data-driven for Mobile Testing tests: Smoke test and build acceptance testing, It is good practice to start with a proof of concept regression testing, synchronization testing, (PoC) on critical platforms before narrowing down compatibility testing, GUI testing and testing to a tool or tool set. One tool may not achieve non-dynamic UI components, as well as cross- cross-platform automation goals. When test auto- platform functional testing. mation tools are evaluated, organizations must Moving Forward not overlook how they integrate with test management environments. One starting point is to The principles and success factors of test identify reusable scenarios across platforms and automation for mobile applications are much group them accordingly. This facilitates the cre- different than those traditionally applied to con- ation of an appropriate, time-saving framework. ventional applications. The attributes of devices, environment, tool sets and test coverage add For native apps, choose critical test cases that can multiple layers of complexity. It requires keen be run across releases as automation candidates insight and a customized approach for a well- and ensure that there is a cross-platform reus- orchestrated automation strategy. ability and multi-browser compatibility. Always keep in mind different form factors while deter- The best practices covered in this paper will mining test coverage, especially for the mobile guide you toward setting test automation goals Web. Consider different connectivity aspects for and choosing appropriate coverage, the right automation execution runs. tool sets and a comprehensive approach. Making automation effective and efficient ultimately In addition, the following recommendations will yields cost and time reduction in the app develop- be helpful in informing your test automation ment lifecycle.

cognizant 20-20 insights 5 Footnotes 1 http://www.gartner.com/it/page.jsp?id=1230413 2 http://www.mediapost.com/publications/search-marketing-daily/edition/2012/07/13/?print 3 IDC Worldwide Smartphone 2012-2016 Forecast Update, June 2012. 4 Ted Schadler, John C. McCarthy, “Mobile Is the New Face of Engagement,” Forrester Research, February 2012.

About the Authors Pradeep Kumar Govindasamy leads Cognizant’s Automation and Mobile Testing Center of Excellence, a practice comprising more than 1,500 experts servicing global customers. He brings over 12 years of experience in the information technology industry and is a valued speaker on topics such as automation, test data management and tools at events worldwide including Starwest 2009 and 2011, Stareast 2011, Swiss Testing Day, HP Universe and EuroStar. Pradeep can be reached at PradeepKumar.Govindasamy@ cognizant.com.

Ramakrishnan Venkatasubramanian heads Cognizant’s Automation R&D and Offshore Mobile Testing teams. He has over 12 years of experience in the information technology industry. His experience spans software development, product engineering, testing, test automation, mobile and technology consulting. Ramakrishnan can be reached at [email protected].

About Cognizant’s Mobile Testing CoE Cognizant‘s Mobile Testing Center of Excellence (CoE) provides state-of-the-art solutions for mobile testing require- ments with best-of-the-breed testing practices and processes. It specializes in offerings such as unit testing, functional testing, multi-language compatibility testing, security testing, automation testing, performance testing, device testing and application testing. Our Mobile Testing CoE features a strong pool of certified mobile testing experts, built through long-term competency development programs. In addition, it houses smart products and frameworks powered by a focused R&D team, which takes mobile application testing to a whole new dimension.

About Cognizant Cognizant (NASDAQ: CTSH) is a leading provider of information technology, consulting, and business process out- sourcing services, dedicated to helping the world’s leading companies build stronger businesses. Headquartered in Teaneck, New Jersey (U.S.), Cognizant combines a passion for client satisfaction, technology innovation, deep industry and business process expertise, and a global, collaborative workforce that embodies the future of work. With over 50 delivery centers worldwide and approximately 145,200 employees as of June 30, 2012, Cognizant is a member of the NASDAQ-100, the S&P 500, the Forbes Global 2000, and the Fortune 500 and is ranked among the top performing and fastest growing companies in the world. Visit us online at www.cognizant.com or follow us on Twitter: Cognizant.

World Headquarters European Headquarters India Operations Headquarters 500 Frank W. Burr Blvd. 1 Kingdom Street #5/535, Old Mahabalipuram Road Teaneck, NJ 07666 USA Paddington Central Okkiyam Pettai, Thoraipakkam Phone: +1 201 801 0233 London W2 6BD Chennai, 600 096 India Fax: +1 201 801 0243 Phone: +44 (0) 20 7297 7600 Phone: +91 (0) 44 4209 6000 Toll Free: +1 888 937 3277 Fax: +44 (0) 20 7121 0102 Fax: +91 (0) 44 4209 6060 Email: [email protected] Email: [email protected] Email: [email protected]

© Copyright 2012, Cognizant. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the express written permission from Cognizant. The information contained herein is subject to change without notice. All other trademarks mentioned herein are the property of their respective owners.