Field Testing Software: Boon and Bane
Some time back I blogged about "Does it Really Matter What You Are Testing?" I still stick by the message in that post about the importance of testing regardless of the product domain. But there is no denying that product development and testing for certain disciplines are more critical and challenging by design with serious adverse impacts in the event of failures—aviation, medicine, and amusement industries, to name a few.
Over the years the technology industry has accepted field testing as an important step in gaining confidence that a product is ready for launch. This is especially true for the disciplines just mentioned, as the consequences of software flaws often turn out to be grave both financially and in loss of life—and the organization’s reputation and global confidence in the industry as a whole.
But is field testing always possible and a boon? Let’s use the aviation industry as an example in our discussion. If we look at some serious aviation failures that have been attributed to software flaws—Ariane 5 rocket failure and Patriot Missiles failure—field testing would have been almost impossible, given the use case scenarios.
Simulated test runs representative of final usage are what testers will need to rely on. The simulators themselves need to be so sophisticated as to expose the system under test to various expected and unexpected scenarios. Often the testers are highly specialized engineers who not only have the software specialization but also the domain specialization.
In some cases, they may have even been in charge of manning the instrument before settling down for a testing career. For example, flight test engineering is a very specialized profession where the engineer may have been a pilot himself.
While there are downsides and challenges in field testing for specific cases, there are other cases where it proves invaluable. The National Transportation Safety Board reports more than one third of air accidents are driven by turbulence. Products to monitor and mitigate these causes will not only reduce accidents but also save big on gas emissions with positive impacts.
A recent post related to this is Delta’s petition to the Federal Aviation Administration (FAA) to allow pilots to use iPads in the cockpit area to help them measure and get very useful information about turbulence. The FAA bans the use of Wi-Fi devices on the flight deck when engines are operational in B737-700/800 aircrafts.
While the FAA’s ban is understandable in this situation, it is equally convincing to see Delta’s reasonings to be able to do this testing and data gathering in the field to take the aviation industry and associated technology to a higher level of safe and economic air travel.
This brings us to the interesting dilemma of the boon and the bane of field testing, which does not have a clear blanket answer. The industry will continue to exercise moderation and due diligence in its field testing, while weighing in on risks and benefits it offers in each case.
