Developing and Testing IoT and Embedded Systems: Questions to Ask
Self-driving cars are the new big thing, and they’re already being developed by many companies. These “smart” vehicles have sensors, computers, embedded software running on many processors, hardware, and communications through the Internet of Things.
The operational and environment scenarios these cars will encounter are practically infinite. Creating software that is smart enough to deal with the operations of self-driving cars in many different situations will take time, a lot of engineering, and proper testing.
How we should develop and test these systems is a big question, and there are no easy answers. For example, do we perform testing throughout the lifecycle, or do we wait until the vehicle is ready for the road? Should we test just functional requirements through verification checking, or should we include “break it” exploratory testing? And should testing be in different environments and employ different use cases outside what the developers define?
I have my own opinions on some answers to these kinds of questions.
First, I think extensive, developer-level testing is essential. I also think we should have internal test labs using modeling, simulation, and automation with multiple levels of software, hardware, and systems driven by risk and math-based approaches.
We should conduct continuous and ongoing field testing and demonstration, both with test teams and users. Car companies have a history of field testing, so development engineers know a bit about it. Field testing confirms any assumptions that might have been wrongly made by developers or during lab testing, and systems monitoring can be done to record data for later analytics.
I also believe we should use in-depth verification and validation efforts to seek high levels of integrity. Introducing independent verification and validation is an added level of assurance that comes from the space industry, which has long created autonomous systems, and it has a proven track record. I think it’s important that these verification, validation, and testing teams be independent and organizationally separate from development.
To do all this, we need highly skilled staff who support verification, validation, and testing and who understand systems, hardware in the loop, software in the loop, testing, communications, operations, modeling, and simulation. We must provide ongoing training to develop these employees and their abilities.
I recommend these starting points over simple confirmation checks. There are few shortcuts when it comes to safety-critical, complex embedded software. We need both good development engineering and testing, and following an agile model will show us what works and what doesn’t by using continuous testing with short feedback cycles. There are certainly other concepts to include in IoT and embedded software testing, but this list gets teams started.
Jon Hagar is presenting the tutorials Test Attacks to Break Mobile and Embedded Software and Testing the Internet of Things and the session IoT Software Testing Challenges: The IoT World Is Really Different at STARWEST 2016, October 2–7 in Anaheim, California.
