Rebuilding Your Test Strategy
Every once in a while I get asked to review the testing process for a company. The request usually comes from someone managing a project that is behind or under some serious time pressure. That person noticed that testing is taking a long time and bugs are increasingly being reported from production. Testing is usually the first thing to look at, because testing work is so focused on finding those problems. I take a holistic approach when I get these requests.
The first thing I do is talk with the testers. They will usually feel like they are being taken to the principal's office, so I make it clear that the goal is to help them work more sustainably and to be more effective. I ask about what a normal day is like, where most of their time is spent, and what they think they need to do a better job.
On the development side, I take a look at what happens in the continuous integration tool, what test coverage looks like at the unit, service, and browser levels, and what happens during the pull request process.
Rebuilding a bad product from scratch rarely works. What you end up with is another bad product, but with different problems. My preference is to spend some time understanding the current process and what testing is happening through the dev process—not what is outlined in a process wiki, but the work that actually happens.
Some of the testing problems I find are testers spending too much time on documentation that isn’t useful, not thinking of their work in terms of test coverage, and lacking skill around test design. I also usually discover that testers are slow because they are finding a lot of bugs, they have to context-switch between projects, they are sometimes separated from the development process, and they usually feel some pressure to build automation. These are all symptoms of a problem, not the real problems. They are caused by something upstream.
If I look upstream I will tend to find flaky tests or entire code repositories that are missing coverage, plus pull requests that are approved without any real review. All of this will eventually cascade down to the testers, making their job slower, more difficult, or both.
Once I’ve looked at the whole process, I make a list of things that could be made better. Maybe we could add some test coverage, automate prerelease testing that is done the same way every time, or help testers get a little better at designing tests that will find bugs. After that, we prioritize the list and decide which thing is most important to work on right now.
Testing is something that happens throughout development, all the way until a code change is pushed into production. It is not an activity monopolized by testers. If you want to rebuild your test strategy, be sure to review the whole strategy.
