Testing Is Insurance, Not Assurance
Another week, another preventable, high-profile tech disaster. The Iowa Democratic Party used a mobile app to pull results from statewide precincts for the Iowa caucus. While there were many reasons why this application failed, the Democratic Party left it to “coding issues.” Anyone with any degree of experience can tell you this excuse really means they didn’t properly test the application.
According to a New York Times report, the app had “never been used in any real election or tested at a statewide scale,” and its use had only started two months prior. The application is even reported to have been released using TestFlight and TestFairy, both less stable models for deploying applications at scale.
As with other large tech failures, like HealthCare.gov or Hawaii’s ballistic missile false alarm, proper testing may not have assured a tech catastrophe was out of the question—but it would have insured that it was far less likely to happen.
Building any application of reasonable complexity requires a complementary amount of testing. While you can never test for every possible scenario, it’s important that prior to releasing software to a user community, you’ve done your due diligence to insure you avert a software nightmare.
Still, when deadlines are short and the pressure is on, the first part of testing that gets cut is release testing. After all, we’ve spent the bulk of our time building and testing a specific feature. But release testing is a critical component of ensuring your product is ready.
While this term can be used differently by many organizations, I define release testing as a broad set of tests against the newer build of an application to make sure that the application works as intended and contains no flaw that would seriously impact the user’s ability to complete typical use cases. In short, it tells you if you’re really ready to release to actual users.
This does not imply that the release testing is assuring that the application is flawless, but that any flaw identified is minor enough to not impact the end-user’s ability to complete tasks deemed as core functionality.
Assuming you can test every scenario a user of your system may attempt is a fool’s errand. We can not always predict negative cases or even how the application rollout may affect user behavior. This is why release testing is your insurance that your release will go well, but it’s not assurance you won’t have a failure.
Like any type of health insurance, you pay for stronger coverage. Your investment in release testing is no different. Strong coverage includes many activities. For business-critical software with many concurrent users, consider performance and load testing, interoperability testing, end-to-end user scenarios, and some simple feature testing specific to the release. If your application has gone untested with a real user community, incorporating user acceptance testing with an actual pilot group is important to ensure you’ve built an application that actually meets the needs of your users.
These crucial tests can help build a sense of confidence that your app won’t be a major release nightmare. Testing can’t guarantee perfection, but not conducting proper testing is gambling with your fate.
