News

Pythian Papers

Case in Point: "The Thursday Rule"

by Steve Pickard

We were fighting fires day and night on a very large and complex environment that has five-nine uptime requirements. Tracking down problems usually took several people an entire day and the problem usually ended up being flawed application code that had been released several days (and several releases) earlier. Since there were multiple code releases per day and no real version control, the DBAs had no choice but to attempt to fix the code in place - and hope they could keep the application and the database up and running in the meantime.

Enter new management with a decree: "Thou shalt only release code on Thursdays at 3pm." The need for firefighting stopped almost immediately. Code flaws are now detected, but not diagnosed, by midday Friday. Flawed application code is simply backed out to the previous week's version until the bugs are uncovered and corrected. Benefits to the client: savings of over 150 DBA hours per month, significantly less downtime (which costs tens of thousands of dollars per hour), and a more stable image for the customers.

Site Map