When Field Testing Isn’t Possible, Use Simulation
The world has been watching India’s latest mission, Chandrayaan 2, to land on the moon. While the effort was largely a success, not everything went according to plan, especially in the last few minutes. The ISRO team lost contact with the lander only about two kilometers from the moon’s surface.
While efforts are being made to regain contact with the spacecraft, Chandrayaan 2 gives software testers a lot of interesting lessons on the criticality of planning a thorough simulated testing effort when field testing in not possible.
Many testers understand the importance of field testing. With all the different mobile versions, in-house testing alone is not going to suffice. A lot of user scenarios, especially driven by the nuances of the environment the application and device will be used in, necessitate field testing.
For example, let’s take testing wearable apps. While core scenarios can be tested in house, field testing becomes important to ensure complete readiness for release. Despite short release cycles and fast time to market, product groups are often including field testing alongside beta testing whenever possible.
However, regardless of the value from such an effort, field testing is not always possible. It could be constrained by several factors, including more common reasons such as cost and time, all the way to unusual and prohibitive reasons, like in the case of Chandrayaan 2.
In such cases, simulation and emulation come in very handy to create a picture that mimics the real scenario.
For example, with Chandrayaan 2, in the drive to imitate the moon’s surface to test landing abilities, the scientists closely worked with geologists to determine where to procure lunar-like soil and rocks within India. This is the level of detail that went into the effort. In most of our cases, the in-house replication for field testing is with a mobile simulator or emulator, mimicking hardware, operating system, network ranges, and more.
Such simulation is not restricted to just functional test areas—it’s being extended to other nonfunctional areas, too. For example, chaos engineering is gaining popularity in performance engineering, where tests are run in a chaos setup with multiple other coexisting applications, similar to a compatibility effort that testers have traditionally set up.
Whether it’s something as basic as a compatibility matrix or as complex as Chandrayaan 2’s setup, simulation should be an inevitable step in software testing. A lot of simulators are available for use, including in the cloud, making access quick and easy. With various options available, the tester’s experience and critical thinking become important to engineer a simulated setup providing a representation of the field that’s as close to the real thing as possible.
