Go-Live Lessons: The Path from Software Development to Production
As a consultant, I interact with a variety of information technology professionals in several different contexts. I like stealing good ideas from one context and sharing them in others.
A hospital’s IT project manager identified an interesting problem. An interface that was supposed to associate lab results with patient records tested fine and was rolled into production. Weeks later, a lab tech noticed that the wrong results were associated with a patient record. No one gets hurt in this story, but you can understand that this was a potentially life-threatening scenario.
Manual checks found that about 5 percent of the results were assigned to the wrong patient. This had not been observed in testing. It had something to do with the system assuming that a timestamp on the lab results would be unique. The assumption proved false under production load when the system bogged down.
The IT manager was perplexed. The go-live criteria had been met, test results were good, yet patient record integrity had been compromised—this was unacceptable. How much testing and review was necessary? Did he need to test production systems forever? How do you budget for that? The suggestion I offered was one I had picked up in a different IT project management context: custom software development performed by an external systems integrator.
On systems integration projects where a vendor is building or configuring a system for a client, you sometimes cross the canyon from development to production and maintenance in several smaller bounds rather than one big leap. This approach seemed like it might be helpful in the hospital’s situation:
1. Establish clear criteria with project stakeholders about what constitutes “production ready” (how much testing should occur, documentation, signoffs, whatever).
2. Once an application is certified production-ready and implemented, retain the vendor and team for a period of intensive production support, sometimes called a warranty period or stabilization period.
3. During the warranty period—typically thirty to ninety days—the vendor and the project team carefully monitor the new production system to assure it behaves as expected, performs well under load, is playing well with interfaces, and isn’t screwing up data. From a contractual perspective, defects logged during the warranty period are fixed as part of the development effort (versus maintenance).
4. Significant defects found during the warranty period reset the timer on the warranty period. There is often a progress payment at the successful completion of the warranty period, and the level of scrutiny is then throttled back to “normal” maintenance and monitoring protocols in the client environment.
My hospital client liked this notion of an intensive post-implementation support period because it was bounded (so he could estimate cost) and because the duration of the warranty and the extensiveness of the production system monitoring could be a point of negotiation with stakeholders.
No one wants quality problems. A warranty period provides language to talk with stakeholders about the level of quality they want and the level of support they are willing to pay for.
This article helped me a lot.
This article is very helpful to me.
This is simply mindf blowing, also help me a lot in my project.