UAT Entrance Criteria: Don’t Negotiate Against Yourself
An important component of any User Acceptance Testing (UAT) plan are the entry criteria: a detailed list of the measures of quality that must be met before UAT can begin. While we can agree that no complex data system will ever be perfect and there might be legitimate reasons to grant exceptions on a case-by-case basis, starting with lax entrance criteria puts the UAT team in a weak position.
When a system is developed by a third-party, User Acceptance Testing (UAT) is a vital quality assurance and contractual compliance step. UAT generally involves the vendor providing their customer with access to a version of a system that the vendor asserts is “production ready” and the customer independently confirming that the system meets the agreed upon requirements.
The “Acceptance” aspect of UAT represents a significant transfer of responsibility for system viability – At the beginning of UAT the vendor asserts the system is production ready – significant issues detected during UAT invalidate that assertion and the vendor is normally on the hook to remedy those defects and try again. If a system passes UAT, faults detected in the future are generally a shared responsibility because the client “accepted” the system.
Recently, a subject matter expert on a client team with extensive development expertise suggested the following entry criteria:
- A minimum of 90% pass rate for the nightly regression test suite
- No outstanding Severity 1 or 2 errors in the bug tracking system (1 = fatal, 2 = significant, 3 = problem with a work around, 4 = cosmetic)
If you are a developer, this might seem reasonable. Perfection is a high and perhaps unattainable standard for complex system development, but from a project management and contractual perspective, I would suggest this standard is too loose and forgiving.
It may be that experienced professionals have been on too many efforts that developed schedule problems. We are used to finding ways to compromise criteria so that we can fudge phase lines, “The system is ‘done’, and it will be ‘finished’ in 60 days” – but sloppy phase lines confound project and contract management problems.
I’m a fan of asking for what you want. As we enter UAT, I would like a system that the vendor asserts is error free. I think entrance criteria should reflect what you want so that the vendor and the client have a clear understanding of what comprises the phase line. If I were writing the entrance criteria, I’d specify:
- 100% pass rate for the nightly regression testing suite
- Zero outstanding bugs in the bug tracking system
Ideally, I don’t want my UAT team wasting time documenting or working around known errors. I want to start with a system the vendor believes is error free and production ready.
Let’s agree that we may need to discuss exceptions when the phase line approaches to preserve the schedule. These should be considered on a case-by-case basis – balancing reasonableness (we may not wish to delay testing because of a few obscure or cosmetic errors) with assuring that the vendor is providing a quality system and not wasting the time and resources of the UAT team if they haven’t.
A challenge arises when the entrance criteria have been pre-emptively lowered. In this situation, the client has already agreed to accept a system that isn’t ready for prime time so resisting further lowering the quality standards can seem petty, “C’mon, you can work around testing those parts of the system until the end of the next sprint.”
Systems delivered to UAT with known faults needlessly waste the test team’s time and energy. They can also undermine the team’s confidence in the system and vendor, leading to organizational change issues – word spreads fast if there are quality concerns, even if they are remedied before deployment.
Don’t negotiate against yourself. Set clear UAT entrance criteria. Expect some minor negotiation as the handoff approaches but make them explicit and thoughtful. Don’t begin UAT unless you are confident that the system is ready to test. Remember, if you don’t have the time or resources to effectively certify that you have a quality system, you blur responsibility for system quality if you accept a system that isn’t ready to test.
