3 Ways to Keep Your Test Suite Lean
A few years back, I was consulting on a project where I encountered a monstrous, constantly failing, long-running suite of Selenium tests. The codebase, which had been around for more than ten years, contained more than a thousand tests, and the full regression suite took eighteen hours to run. Worst of all, the tests failed 100 percent of the time.
The situation was grim, but after a lot of hard work and persistence, we turned the test suite around by focusing on three qualities for our tests: They should always be valuable, reliable, and fast.
1. Make Tests Valuable
One big problem on this team was a culture that rewarded quantity of tests over quality. The thinking was, the more tests you wrote, the better a tester you must be. Unfortunately, over many years, the number of tests grew out of control.
The team started with cleaning up the test suite and only retaining tests that were valuable. It was important to understand that automation should add value, not be done just because. Keeping this simple rule in mind will have a huge impact.
Here are a few other tips for creating valuable tests:
- Re-evaluate and analyze so that each test's purpose and level of focus is clear
- Collaborate with your team on testing so that everyone is aligned on the value each test delivers
- Emphasize quality over quantity—write tests that test the right thing; don't just test everything
2. Make Tests Reliable
Another big problem on this team was that tests constantly failed. Tests should have a consistent, passing result every time they are run. This sounds straightforward, but sometimes unexplainable behavior happens to tests, causing distrust.
If your team doesn’t trust the output of its tests, then those tests are useless. It’s critical that tests have consistent passing results.
Here are some tips for creating reliable tests:
- Separate tests that don’t have consistent results, fix them, and put them back one by one as you get them to pass
- Rerun failures automatically as a failsafe and to avoid having to run the entire test suite over again
- Use specific test selectors to help avoid CSS changes that break tests and indicate that the element is used for testing
3. Make Tests Fast
As any test suites grow, the tests will take more time to finish, and build times will lengthen. The longer the tests run, the longer the feedback loop. That's bad, because when you’re waiting, you're not shipping code. Whenever possible, look for opportunities to make your tests run faster.
Here are some tips for creating fast tests:
- Parallelize tests so you can run multiple test classes at the same time
- Avoid testing in the UI, and look for opportunities to automate features at a lower level (API, integration, or unit)
- Run your tests headless to avoid having to wait for browser pages to load and render
Start Optimizing Your Tests
By following these three principles, we were able to whip the test suite into shape. While the process was tough, I’m happy for the lessons I’ve learned.
Meaghan Lewis is presenting the session Keeping Your Tests Lean at the Agile Testing Days USA 2018 conference, June 25–29 in Boston, MA.
