Continuous Testing Is Not Automation
Many people confuse continuous testing with test automation. That makes sense, because you cannot do continuous testing without automated tests. But it is much more.
There are four basic questions you can ask to see if your automation is truly continuous:
- Are your tests automated in real time?
- Are they included as part of your continuous integration (CI) pipeline?
- Do you allow them, alone, to determine that a version of code can move to the next step in the process?
- Are performance and security tests included in 1,2, and 3?
If you cannot say yes to all four questions, then you are not doing continuous testing.
So, how do you go from automated tests to continuous testing?
First, try to run your test suite back to back. If it takes more than a minute or two for each test to run, you need to look at the tools you are using for automation. Are you able to automate in real time without a complex abstraction layer for the testers? Are you using a third party that is inherently slower than an open source tool? Your answers might indicate that your automation tools do not support continuous testing.
The other reason tests could be slow is if they all originated from your regression suite and thus are all end-to-end UI tests. In this case, you will need to adjust how you have applied the testing pyramid to ensure you reduce the number of tests, so that more is done at the API and component level.
Second, look at the failed tests to see if different tests failed in both runs. If a test passes in one run and fails in the next, then you might have flaky tests, which again can be solved by refining your testing approach, as well as evaluating your testing framework for good coding practices.
Flaky tests could also be caused by dependencies on data or downstream systems that are not reliable. In these cases, you should pull out all the data conditioning into its own suite of tests, or find a more reliable way to inject data, like via a test data management tool or through APIs directly. You also might need to leverage a service virtualization tool to replace unreliable or unavailable systems.
Third, integrate the suite with your CI tool. You could start with deconstructing your suite of tests so that you have multiple suites running at different points of the process, each building on so that coverage increases. You will also need a dashboard so teams can see the results of the tests and gain confidence that a failed test is truly a defect.
Finally, integrate performance and security tests into this process. You can follow the same steps as above, because the tool, approach, and CI integration may all have to be addressed.
Continuous testing has a higher-level maturity than automated testing that could require a totally different way of working. The result is a fast feedback loop for developers—and a faster path to production.
Adam Auerbach is presenting the session Continuous Testing Is Not Test Automation at Agile + DevOps East 2018, November 4–9 in Orlando, Florida.
There's a better way to solve the flaky test problem, a way that doesn't require introducing quality risk by hiding parts of your IT ecosystem (dependencies). But, it requires a re-engineered solution to your automation, so not compatible with your tools.