5 Steps to Stable Continuous Testing in DevOps
Achieving optimal automated testing with thorough coverage can be difficult for many teams.
The first step to resolving the test automation roadblock is to understand the three types of personas in testing. Each persona demands different technology and methods.
A business tester is a stakeholder with less technical skill. Business testers implement mostly manual and exploratory testing and often leverage codeless testing.
A software development engineer in test (SDET) is knowledgeable in coding and testing. They can create functional and nonfunctional code-based test automation scripts with a test automation framework.
The software developer creates unit and build acceptance tests and operates in code-based testing. Developers seek fast feedback from their code changes through their testing.
With this knowledge of testing personas, you can begin these five steps toward achieving stable continuous testing.
1. Stable automation
Automation is stable when all testing issues have been resolved and smart reporting clearly discerns the false negatives from the real issues.
Your team’s skill set will be a major factor. If your team includes developers with experience in code-level scripting, then using a platform like Selenium or Appium is suitable. If your team is business stakeholders with less technical skill, a codeless or low-code test automation tool is a good target.
Spend time ensuring that true stability is reached, as this phase is the foundation for the continuous testing process.
2. Daily cycle
The next step is to establish a daily cycle of test automation. Add a limited set of scripts into a repetitive pattern that runs at least once a day automatically on multiple platforms, with tests applying to both mobile and web.
You will know your process is ready for the next step when your test scripts run robustly with little to no false negatives and your organization trusts that test automation is working. This buy-in is crucial as you progress to full continuous testing.
3. More coverage
While test automation is a time-saving activity, it is also a gateway to more thorough test coverage. The overall goal is to reach meaningful functional and nonfunctional test automation coverage.
Anything not automated must be done at the end of the cycle, which can drag down release cycle speeds. In this stage, focus on a low-maintenance but modern lab that can scale to handle many executions.
4. 90% coverage
As your coverage increases, strive to reach about 90%. You will need additional layers of environment control and capabilities such as network virtualization, service virtualization, and sensor mocking, as well as high-trust testing to run from your CI, which will enable mature continuous delivery.
At this stage you can expect dozens of executions to run at once, so analytics will be indispensable. Your reports must encompass millions of lines of actions, a massive amount of data, screenshots, and videos. Also watch for ineffective reporting solutions that tend to break.
5. Continuous testing
At this point, automated testing will reliably run every day upon every code change, with very detailed insights into the entire pipeline. Your DevOps will now benefit from faster feedback, more accuracy, and reduced business risk.
Continuous testing ensures DevOps has the tools to deliver quality, modern code that is ready for the future, so be sure you understand the key stakeholders and progressive steps.
Eran Kinsbruner is presenting the session Stabilizing Continuous Testing in DevOps at STARCANADA 2019, October 20–25 in Toronto, Ontario.
