Testing Lessons from Toyota and Nissan Recalls
Toyota motors recently announced the recall of approximately 1.9 million Prius models, and Nissan was also in the news for similar reasons. Common to both of these cases is the underlying reason for the recalls—software programming errors in some of the components. What are the potential testing lessons we can learn from these recalls?
Understand the importance of corner case scenarios: A corner case is a problem that only occurs outside of normal operating parameters—specifically one that manifests itself when multiple environmental variables are simultaneously at extreme levels—even though each parameter is within the specified range for that parameter. Below is a description of the sequence leading to one of Toyota's recall defects:
The software’s current settings could result in higher thermal stress in certain transistors, potentially causing them to become damaged…In rare circumstances, the hybrid system might shut down while the vehicle is being driven, resulting in the loss of power and the vehicle coming to a stop.
As with many automobile recall stories, this one didn’t represent a straightforward failure. This was somewhat closer to a corner case definition. One of the arguments pertaining to corner case defects is that unless the application under test is mission-critical, like air traffic control or cancer chemotherapy, it should not be given more importance than needed.
The book Lessons Learned in Software Testing: A Context-Driven Approach states that testing is done to find information. If a project is like a road trip, testers are the headlights of the project. Uncovering corner cases should be given proper stature in the test plan in alignment with the project's scope and overall mission of testing. As an example, Google relies on Dogfooding to unveil the corner case bugs.
Give due respect to intermittent bugs: The key issue leading to the recall of Nissan vehicles was that the software triggering an air bag launch had faulty logic. In some cases, the software incorrectly stated that the passenger seat was empty when it was occupied. Luckily, this error didn’t cause any fatal accidents.
This bug probably didn’t occur every time the software logic was tested; otherwise it would have been found much earlier and would not have lead to a recall. Intermittent or non-reproducible issues are indeed a bane. The fact that a bug is seen even once but not seen again in many iterations of tests would prompt a tester to report the bug—despite the risk of being sent back by a developer.
Such bugs usually carry a significant cost of investigation, but depending on the severity of bug, the effort is worth the time spent as the root cause may point to larger instability in the system. A blog post from Google on minimizing unreproducible bugs provides some valuable tips for developers and testers.
What lessons do you take away from these cases?
