Continuous Testing, Continuous Variation
With the arrival of continuous integration/continuous delivery (CI/CD) the notion of continuous testing (CT) is taking center stage. Knowing that comprehensive tests are running smoothly can be of great benefit for the CI/CD pipeline. But running tests can be both time and resource consuming, not to mention that tests can become boring and rigid. Using the repetitive character of CI/CD for testing can be a way to address this.
Reducing the Number of Test Runs
In a well-run CI/CD process, rebuild times will depend on how many components are affected by changes and need to be rebuilt, including execution of their unit tests. However, the amount of functional testing needed when there is a rebuild is less straightforward. Functional tests, in particular business level tests, can have a wider range than a single component and may need to run in multiple configurations and/or environments.
The need for many test executions can be addressed with technical means, like more hardware, headless browsers, API testing, etc. However, an additional interesting approach can be to leverage the fact that tests are executed so often. Instead of running the same test over and over again, bring in some variation. I like to call this "continuous variation," as a next step up from continuous testing.
The first way to embark upon continuous variation is in the choice of which tests to run. Obviously tests that address the components that were changed would always be executed, but to gain more coverage, pick additional end-to-end tests, either randomly or in a round-robin fashion. That way, all tests get their chance to run and find possible side-effects of changes.
For configurations and environments, a similar principle can be considered. Use a list of configurations, select or provision one or more configurations and/or environments from that list, each time a test run is due. The list could be compiled or generated using pair-wise testing. For more information on pair-wise testing, see www.pairwise.org.
Making Tests Less Boring and Rigid
Continuous variation can also help address another issue with automation: its tendency to make tests rigid. Once bugs found by the tests have been resolved, the automated tests can have a hard time finding any new issues.
However, knowing that tests will be run many times makes it possible to follow a data driven approach. Instead of hard-coded input values, use variables for which the values are picked from a row in a table, which then can include expected outcomes. In a traditional data-driven approach, a test will loop through all rows, but in Continuous Variation it will only take one row each time the test is executed. No looping takes place, since that is already taken care of by the repetitive CI/CD process. This applies even more if the variation is done in multiple places in an end-to-end test, since in the course of time many combinations of values will be used.
Leveraging the continuous repetitions that are part of the CI/CD process can be one way to reduce testing times while increasing their effectiveness. If and how this can be meaningful can be very different from case to case and is best for the DevOps teams to decide upon.
