Different mobile challenges need different mobile testing solutions

When selecting a test automation solution for your mobile applications, it quickly becomes clear there are many decisions to consider:

• Do I need a real/physical device cloud?
• Would I be better off with the flexibility of a virtual device cloud?
• Or should I assemble my own test automation solution from components?

With so many choices, which one is right for my team, applications, and business? 

There are many potential approaches to mobile testing. The right choice depends upon your specific needs and impacts the level of effort, time you spend, and even final app quality. Understanding the benefits and limitations will help you select the right option.

Real devices

Organizations turn to real devices believing they provide the most realistic test environment for their mobile applications. After all, they are the devices that users actually use. However, there are several important considerations when moving towards testing with physical devices. Testing across real mobile devices means either owning or having access to these devices. Buying and maintaining a large set of devices quickly becomes prohibitively expensive. This expense repeats itself whenever you need to upgrade these devices to the latest model (e.g., the newest iPhone). As a result, testing using physical devices may involve using a limited set, reducing your coverage for the devices in use by your customers.

Real device clouds (grids) provide access to different devices and configurations located in different geographic regions when needed. Real device clouds have the benefit that they eliminate the need to purchase and own each device, and you typically can use a wide range of devices. However, real device clouds also come with some potential drawbacks. The biggest issue is that real device clouds are stationary and stored in an IT room, which is in stark contrast to actual usage. Users access their mobile phones in cities, out in nature, on the subway, and at a near-infinite number of locations. Device clouds contain real devices in a highly artificial environment.

How can device hardware features, such as cameras, accelerometers, and 5G be realistically tested as part of a device cloud? The answer is they can’t.

Realistic testing of device hardware features requires manual testing with a physical device in hand.

Virtual devices

Like real devices, virtual devices provide access to a wide range of mobile devices using emulators and simulators. Virtual devices can also be run locally or accessed through a provider as part of a virtual device cloud. The key difference is that there is no real device needed or used when testing on virtual devices. Selecting a specific model of phone is part of the virtual device configuration, which can be reconfigured when needed to different models, operating systems, or applications quickly and easily.

One benefit of using virtual devices is testing is faster when compared to real devices. With real devices, you have a limited resource — the devices. If a particular phone or tablet isn’t available, your application testing must wait, leading to longer release times. With virtual devices, there is no longer a concern of whether a particular device is in use or not. You have the flexibility to run tests on essentially any device, and even on several in parallel as supported by your test environment. Also, preparing a new real device before testing and tearing it down afterwards can be time consuming and costly. This time adds to overall test time and the total time to release an application.

Even after setting up a real device, there is no guarantee it is in exactly the same state as the last time you used it. These state differences can lead to inconsistent, hard-to-debug test failures that are unrelated to any code changes, draining your team’s time and energy. Some examples of real device state include settings, permissions, cached files, stored assets, and many others. Virtual devices don’t have these issues and provide ensuring a “clean slate” between tests.

The clear drawback of virtual devices is that they aren’t, well, real. Testing of hardware-specific features on virtual devices must be mocked or faked in some way (e.g., images can be injected rather than using an actual camera). While this fabrication can often be valuable in testing hardware interactions with application software, it is less accurate than using a real device. It is important to note that real device clouds have these same issues as keeping a device stationary in a cabinet restricts using hardware capabilities. Testing hardware device capability typically requires manual testing (in hand) using a real device.

Virtual devices offer greater flexibility, but with many of the same limitations as real device clouds. 

Combining for a better, balanced solution

Clearly, virtual and real devices each have advantages and disadvantages with neither being the best solution for all scenarios. In fact, Apple and Android both provide guidance regarding how to use both with virtual devices being used for most testing and debugging that doesn’t require real hardware. Our recommendation is to mix test execution solutions based on specific testing needs.

• Manual (handheld) testing on real devices is ideal for limited testing of device-specific hardware capabilities (e.g., camera, 5G, accelerometer). This approach allows for realistic device usage in different locations. However, it can be expensive to purchase and maintain all needed devices and cannot be done at scale.
• Real device clouds address many issues present with manual testing. They eliminate the need to purchase and maintain devices yourself and may be lower cost. Also, real device clouds automate test execution, providing a scalable approach. However, keeping devices in a lab cabinet limits the testing of hardware capabilities — which is a main reason to use physical devices to begin with. Also, using real devices potentially adds extra time due to waiting for a needed devices and system configuration.
• Virtual device clouds give the most flexibility, providing access to a wide range of devices using emulators and simulators. They also make it easy and fast to reconfigure and ensure a clean execution environment across tests. Virtual devices require a level of mocking of device hardware. However, not all tests may use these hardware capabilities or require real devices. With their flexibility and ease of use, virtual device clouds can serve as a test execution “workhorse” for many application tests.

Organizations benefit from a mix of real and virtual devices. Real devices (specifically with handheld, manual testing) provide realism when testing capabilities require device hardware. Virtual device clouds provide the greatest ease of use and flexibility for tests not using device hardware capabilities. 

Building a complete environment

Selecting a test execution environment is just one part of your overall test automation. In addition to test execution, a complete test automation solution includes test authoring, test management, version control, root-cause analysis and reporting, and other essential components. Often, each test automation component may focus on a specific area while lacking capabilities in another. For example, a best-in-class real device cloud may only provide test execution while lacking strong capabilities to author tests.

As an example, many organizations use open-source, coded test authoring such as Appium. However, Appium does not include other portions of test automation, such as test execution devices or clouds or test management. Organizations would need to build and integrate these components on their own as part of a complete test automation solution. Likewise, choosing a device cloud first may often mean selecting and integrating a separate test authoring tool.

It is important to note that different components should be used based on the needs and skill level of the team. Appium is a coded solution, and testers need to have coding skills. If reducing maintenance is a goal, you should consider a solution with strong element locators and self-healing capabilities. Also, with the growing use of AI, top-of-the-line AI capabilities provide strong efficiency benefits to your team.

Building a complete solution from parts takes additional time and effort and may result in a solution that is less complete than one purpose-built for test automation. 

Summary

Organizations must consider mobile testing as a whole when choosing a test automation solution. It is important to decide where to use virtual or real devices based on your applications and the needs of your team. However, test execution is just one component with test authoring, test management, version control, and overall integration being essential to any test automation strategy.

Tricentis Testim Mobile: Complete mobile testing for your mobile challenges

Testim Mobile brings together all the components for complete mobile test automation in a single solution. Testim Mobile enables teams to create stable tests faster through its low-code test authoring and seamlessly execute them on its Virtual Mobile Grid. Proven, proprietary technology allows Testim Mobile to more completely understand the structure of your applications for better and more-complete testing. Where other approaches struggle, such as with hybrid and cross-platform applications, Testim Mobile provides stable testing.

Testim Mobile provides industry-leading capabilities to help your team deliver higher-quality applications:

• Faster test creation through AI-powered, low-code test recording
• Greater test stability by locating multiple elements, even when applications change
• Unmatched application support for native, hybrid, and cross-platform React Native and Flutter applications
• Seamlessly connected test authoring and execution entirely in the cloud
• Built-in test management, root-cause analysis and debug, and integration with your release pipeline

Interested in learning more? Register for the webinar on Feb. 6.