Before You Buy That Test Tool, Answer These Questions
Tools are a normal part of testing jobs because they can amplify our ability to learn about product quality. I have regularly had to review new tools for automation, performance, or monitoring to see if some solution will help us test better.
Before we even look at tools, though, there are two questions we should ask.
What Problem Are You Trying to Solve?
This is the most important question you can ask.
I sometimes see people starting from a wrong premise—a common answer is, “I want to automate our browser tests.” My reply to that is usually something like, “Cool, why do you want to do that?”
A more reasoned answer I would like to hear is more along the lines of, “We are currently using an expensive crowd-sourcing company to do a lot of our testing. They are working from scripts and occasionally find something, but they never deviate from the script. I think it might make more sense to have a computer perform testing that is done in an identical way every time. We might even be able to use some compatible programming libraries to make the tests a little deeper.”
This answer shows that you have identified a financial and technical need for a tool.
What Process Will You Use?
What will you do with the tool once you have it?
Whether the tool is free and open source or commercial, your company is going to sink money into it in terms of people and hardware resources. You also probably really want to solve that problem we just talked about.
You’ll want to think about how you will use the tool before you settle on which one you use. Who will work the technical parts? Who will consume the information the tool produces? Who will act when the tool triggers you to look closer at the product you are testing?
A few years ago I was working on developing some test automation for my employer. My manager had asked for a set of tests to be automated so we would have a release smoke test. The request sounded easy enough. I wrote the tests, went through code review, got the tests checked into our code repository, and figured I was all done.
But the tests never ran—and we didn’t find out for three months.
No one on the team was monitoring the tests to see if they were passing or failing after a deploy, so we never noticed that they weren’t running. We had a silently failing misconfiguration in the virtual machine where our tests were supposed to run.
Deciding who would be monitoring the tests before we started, at an absolute minimum, would have saved us several months of lost time.
Feel Free to Ask
Tooling can be wonderful, fun, and helpful. But remember to think carefully about it before you are committed. Make sure to talk specifically about the problem you want to solve and how things will work in practice.
