Mobile Testing: Simulators, Physical Devices, or Both?
Do we really need physical device testing for mobile products?
The answer is yes and no. It depends on what aspect of quality you want to validate and where you are in the product lifecycle.
For any mobile product, there are many ways to simulate the mobile experience. For a mobile website, we can build a test hook within the product to show mobile experience, use desktop browser mobile simulation, or use a virtual device, such as a simulator or emulator, for the specific platform. We can also use virtual devices to test the behavior of mobile apps.
Simulation has many advantages over physical device testing. First, it costs much less to build and use simulators and emulators compared to physical devices. Simulation only requires desktop hardware and often can have multiple instances running, while physical device testing demands purchasing the device, Wi-Fi or a reverse-tethering network setup, and only a single test instance can be executed on each device. Furthermore, the maintenance cost is much higher for physical device infrastructure due to software and model updates and potential device issues.
Second, test speed using a simulator has gotten much faster. Simulators used to tend to be a bit slower, but with a solid-state drive and good GPU, they can have comparable speed (or faster) compared to testing on physical devices. Based on my company’s data, desktop browser simulation can run the same test on the same browser in less than half the time it takes on a physical device.
And last, simulation is more stable and easier to scale up. It is much easier to recover any issue from a virtual test through a restart or reinstall, whereas on a physical device, there are times when manual intervention is required. There is also very little labor cost to scale up the simulation setup once you have a solid unit built.
However, do not completely discount physical device testing yet. We still need it for the areas simulation cannot cover. For instance, we are not testing 100 percent user experience through simulation. We have seen some issues, such as problems with the user interface or an app crashing, that could only be reproduced on a physical device. There are also features that are not supported on virtual devices—for example, voice input is not supported on iOS simulators. And due to hardware differences, some test types are required to be executed on physical devices, such as performance and battery life, to get more valid and consistent results.
Mobile testing with simulation is cheaper, faster, more stable, and easier to scale. But testing with physical devices is closer to a real user experience and can have better test results in some areas. So, when you’re testing mobile products, it’s best to heavily use simulation in the earlier part of the lifecycle to support continuous integration, and use more physical device testing when the product enters the stabilization and prerelease phases.
By using a blend of both simulation and physical devices, you’ll get the most comprehensive and most accurate mobile testing coverage.
Danni Wu is presenting the session Integrate On-Device Test Automation into the Dev-Release Pipeline at the 2016 Mobile Dev + Test conference. Learn more at https://mobiledevtest.techwell.com.
