Testing at 43,000 Feet: Reporting Risk That Matters
When I was testing a new Boeing 777 simulator, I knew I would spend a month in "the box" with a couple of binders full of test procedures. This was an exhausting job due to irregular schedules, plus the concentration required to fly the plane (often in bad weather, with one engine out and another severe system failure) while following the test procedure and filling out the online test results forms. I also had to look at electrical schematics, mechanical drawings, or the code when something did not work as expected.
How do you test several million lines of code, not to mention a cockpit made of kilometers of wires all connected to mechanical parts, motors, speakers, and projectors, enclosed in a moving platform with a 220-degree visual field on a laser-adjusted screen fed by a powerful image generator?
The output of that work was numbers. For example, the electrical system was 80 percent pass, 10 percent on hold, and 10 percent fail, with issues created in Jira. The status of all aircraft systems (hydraulic, electric, fuel, pneumatics, motion, visual, sound, emergency power, etc.) had to be reported the same way.
Some test frameworks are very good at providing numbers and charts. Just download your open source tool, run it, and voila: A couple of hours later, the first graphs are visible. But what does it all actually mean?
My life changed the day I swapped my outlook from "What do I have to test today?" to "Why do I have to test today?"
That simple change triggered a flow of questions about why I spent almost ten thousand hours of my life in a cockpit, testing and checking. Why do I test a landing gear selector manufactured a couple of weeks ago in our facility? Why do I test that electronic box coming from an on-board equipment manufacturer? Why do we do static code anaylsis on third-party software that has been used on actual airplanes for many years?
It turned out that we were so used to dashboards and numbers that we had forgotten the prime objective: to raise up issues that would prevent qualification of the device by aviation authorities. Numbers look correct and graphs are comfortable to read, but do they really provide the information we should be looking for?
Automation helped me to get rid of some trivial activities, but now, the numbers are disclosed only if they are necessary, and they have to be provided with some statement. This helps keep us on track and sure we are testing for the right reasons.
After all, autopilot can land a plane with less than fifty meters of forward visibility above the runway, but if autopilot fails, who has the skills to land that plane in such conditions? Similarly, if test automation fails, who knows how to execute your necessary tests manually?
Alexandre Bauduin is presenting the session Testing at 43,000 Feet: Reporting Risk That Matters at STARCANADA 2018, October 14–19 in Toronto, Ontario.
About the Author
Alexandre Bauduin is a 51 year old world traveler. He worked in consulting firms gaining experience in several fields (medical, manufacturing, aerospace, pay TV, data warehouse—to name a few) in different countries (Switzerland, France, Spain, Canada, etc.) His career started in the space industry where he discovered his passion for aerospace, working on both military and civilian projects. He was sometimes steered away from aerospace but his passion pushed him to become an airline pilot, as a way to really understand how those instruments he programmed and integrated were operating in a cockpit. One of his last challenges was to organize flight simulator testing into a lean manufacturing environment. He works with milling machines, draftsmanship, accounting and finance, software development, electronic design and industrial robots, and it is always fun for him to use an oscilloscope, an ARINC bus analyzer, and step into assembly language or stall a Boeing 777!