White Paper
Mobile Emulators vs. Real Devices
Author - Kelvin Kam
Overview Mobile apps and mobile internet usage is growing exponentially. As more and more companies roll out mobile business-critical applications, such as mobile banking, commerce or health apps, their risk exposure is also on the rise. Alongside the increasing demand from end users for fast service and an excellent user experience, organizations are also busy ensuring compliance with security and privacy laws and regulations. Accordingly, companies require mature mobile testing solutions that allow them to deploy and support their mobile application development in a timely and cost-effective manner, while reducing their risk exposure.
Emulators are available and are widely used for both manual and automated mobile application testing. They are powerful tools for developing mobile applications; they have unique features which enable them to provide a rich set of development tools as well as an integrated debugging environment and in most cases are available for free from the vendors. They fall into three main categories:
Device emulators - These are generally provided by device manufacturers and simulate the actual device. Device emulators take the place of the actual hardware and can provide an incredibly beneficial test environment.
Browser emulators - These simulate mobile browser environments. Whilst useful for determining the functionality available in a particular mobile browser, they are useless for device-specific testing.
Operating System Emulators - Microsoft provides emulators for Windows Mobile, and Google provides an emulator for Android. These run within a simulated mobile device environment and provide access to applications running within the operating system, e.g. a Web browser.
However, by definition, emulators are not identical to the real target environment. This happens in the world of aviation, where pilots are trained on emulators to test situations that they cannot simulate using real aircrafts. What is the difference between an emulator and a simulator? Emulation is a process of mimicking outwardly observable behavior to match an existing target. The internal state of the emulation mechanism does not have to accurately reflect the internal state of the target which it is emulating. Simulation, on the other hand, does involve modeling this underlying state. The end result of a good simulation is that the simulation model will emulate the target which it is simulating.
Since mobile applications are used on real handsets and not emulators, clearly the closer you get to the actual platform during the QA process, the better quality you will achieve. On the other hand, using only
White Paper real devices for development and testing may not make sense for every application. The trade-off in terms of convenience and cost versus the need for risk management and quality must be carefully considered in the testing strategy. Obviously, if no budget or logistical factors exist, using a real device is always a preferable option for testing.
This article will highlight the pros and cons of testing mobile applications on using emulators versus real devices.
Advantages (Mobile Emulators) The most obvious advantage is price. In most cases, mobile emulators are completely free. All you need to do is download the software, install on your PC, and you're ready to go.
In addition, since emulators are simple client software that run locally on your PC, they run faster and with less latency than real devices connected to the local network or in the cloud.
The emulator is usually part of the SDK provided to developers. Due to their integration with the development environment, mobile emulators provide the developer or tester with access to detailed information such as debugging information which is very important for the development phase. This allows for convenient step-by-step debugging of your application on the emulator.
In addition, there may be cases where mobile emulators can give you the benefit of simulating hard-to reproduce scenarios (low battery, certain GPS coordinates, etc.) that a real device connected to a network cannot easily support. However, this simulation is incomplete at best, as there is no guarantee that real devices would behave the same in such cases.
Disadvantages (Mobile Emulators)
False Impression Even if the testing goes perfectly, you cannot be 100% sure that your data can actually apply to a real device. This raises the question of which tests need to be double-checked on a real device and which can be assumed reliable on the emulator. Also, in the event of a test that fails on the emulator, testers need to decide whether to perform the test on a real device or simply assume that the function does not work and requires correction.
Hardware and Software Differences The emulator is typically a “plain vanilla” version of the OS and often does not reflect the specific hardware and software features of each supported device. For instance, Google releases the Android emulator software (let’s say 2.2) which is then delivered to Motorola, HTC and Samsung. Each of those vendors needs to make an impact on the market, either in terms of hardware or software. With respect to the former, each vendor will try to offer more appealing hardware features, which in many cases were never tested against the “plain vanilla” version of the software. In addition, to make their device more attractive, each vendor will attempt to enhance the "plain vanilla" software (e.g., faster browser,
White Paper better launcher, more efficient memory management, etc.)
Whether those enhancement work well with your application is another story…
With companies such as Apple, the emulator is a more accurate representation of the device since the software and hardware vendors are the same company and basically have the same goal.
Different Network Environment In terms of network configuration, mobile emulators run on the PC, connect to the LAN and access the Internet via your personal/corporate firewall. Using real handsets, the network is connected to the radio interface and from there to the Internet. In many cases, the low level network stack itself is completely different. Regardless of whether this is good or bad, the network environment is different and will result in your application behaving in another way.
Differences in Computing Resources As compared to mobile emulators running on PCs, real handsets have limited resources. The mobile handset is a compact platform with limited CPU power and memory. An application running on a PC does not accurately reflect the user experience on your handset, which, in extreme cases, it may not even work on at all.
Mobile networks affect application behavior Since the mobile device is still a phone, network-related events (e.g., incoming call, text message, etc.) must be tested to determine their impact on your application. This is very hard to accomplish using an emulator, because quality of the network varies between carriers, states, countries, and regions. Since emulators are not connected to the mobile network, they are not capable of providing an answer for how these will affect your application.
Advantages (Real Devices) Testing on real handsets always gives you accurate results. There is no need to worry about “false negatives” or, more importantly, the ever-dangerous “false positives.”
Real handset testing is typically performed in a live network. This is vital because network-related events could affect the quality of your application. What happens, for instance, when a call or text message comes into your device in the middle of a transaction? In addition, different network technologies (e.g., HSPDA, UMTS, LTE, WIFI, etc.) could impact the way your application behaves.
Testing on real devices is the only way to truly understand the user experience, taking into account the CPU, memory, screen size, etc. for a given device. When it comes to the quality of your services, there is no substitute for using a real handset operating in one or more live carrier networks.
Furthermore, it is much easier to expose performance defects with real handsets, as well as defects which are the result of the handset itself or its environment. These types of defects cannot normally be
White Paper detected using emulators.
Disadvantages (Real Devices) As stated above, real handsets and tablets are physical resources which need to be managed. The logistics and costs involved in procuring and managing these devices are significant. How many different types of handset do you need to test? Which one should you pick? This is a question that can be very difficult to answer, because there are so many handsets/tablets in the market.
In the initial development stages, real handsets are harder to connect to IDE than emulators, which can slow down the debugging process. However, this requirement is less relevant in subsequent testing phases.
Conclusions We have seen both the advantages and risks of using emulators for mobile application testing. We have also seen the need for testing on real devices in order to ensure high quality with respect to network issues, location-based services and true user experience. Thus, when it comes to deciding between emulators or real devices, it would appear that the optimal testing solution for companies is not "either/or" but rather a combination of both.
The decisions of when to use emulators or real devices, and to what extent, are a function of each organization's risk management approach. Obviously, there is a trade-off between the cost of mobile testing and the risk of releasing a faulty application to the market. In the case of a mobile banking application, for example, a software defect may result in severe business consequences. On the other hand, an independent developer that can afford to take risks may be willing to test only on an emulator and save costs.
The grey area between these two extremes is likely to vary among different organizations, depending on each one's risk management approach, internal needs, and customer demands. Each organization will need to determine at what stage to introduce real devices, how many devices are sufficient to cover market needs, and how best to manage those devices.
References http://mobiforge.com/testing/story/a-guide-mobile-emulators
http://h30499.www3.hp.com/t5/The-Future-of-Testing-Blog/Emulators-vs-Real-Devices-for- Mobile-Application-Testing-The/ba-p/5506993
http://www.klauskomenda.com/archives/2008/03/17/testing-on-mobile-devices-using- emulators/
http://mobiforge.com/testing/story/testing-physical-devices-made-easy