Is All Testing Exploratory? A Slack Takeover with Michael Bolton
Thought leaders throughout the software community are taking over the TechWell Hub for a day to introduce themselves, answer questions, and engage in conversations.
Michael Bolton helps people solve testing problems that they didn’t realize they could solve. He is a well-known speaker in the industry and has more than twenty-five years of experience testing, developing, managing, and writing about software. Bolton presided over our most recent Slack takeover, which led to some insightful discussions.
Is All Testing Exploratory?
“How do you report exploratory testing? Exploratory testing is sometimes more convenient to point out the distinction between testing and checking.” —@Radovic Ognjen
@michael.a.bolton argued that all testing is exploratory and that checking can be a piece of that exploration. He then introduced a simple analogy: Just as chopping can be done mechanically within cooking, so too can checking be done mechanically within testing.
Bolton then said that, in cooking, tasting and deciding when to taste cannot be automated, nor can figuring out when to use a knife, a mandoline, or a food processor. We use these tools to help us chop, like we can use tools to help us test, but it’s important to make the distinction that machines don’t do the work; they help us do the work.
Tools to Assist with Automating Checks
“Are there tools that seem promising to assist with automating checks which you might suggest are worth further exploration and study? I admit that I have a personal bias towards always seeking the ‘silver bullet’ that will make everything better. Haven't found it yet, and likely never will, but I've learned many things while searching for it.” —@Mark Waite
There isn’t a one-size-fits all tool for automating checks. To find the tool that will help you, Bolton suggested starting with asking, “What problem am I trying to solve?”
Bolton said he finds that tools help with checking for certain kinds of shallow problems and for amplifying our reach, vision, and volume for certain kinds of deeper problems. Ultimately, however, the tools testers use to find those problems aren’t important as long as they help testers do their jobs.
We Can’t Show That the Product Works
“I compare the gatekeeper role to stopping a train. As gatekeeper I stand on the tracks, hold my hand up and say ‘stop.’ The manager at the control decides whether to run over me. It's the difference between testing to show it does what was intended and testing to see whether it works.” —@Doug Hoffman
As Bolton said, "We. Cannot. Show. That. The. Product. Works." Rather, testers can show that they’ve observed a problem, or they’ve observed that a product appeared to work in some circumstances.
He suggests there are two things testers must do:
- Find out how the product is inconsistent with the requirements documentation
- Find out how the product is inconsistent with the actual requirements and with anything else that someone might consider desirable or important
