Production Advice for Developers (3 short pieces)

Posted in: Technical Track

I’ve spent over a decade managing various production systems. After spending so much time with systems that are mostly not working as they should, one develops certain outlook on life. Like the deep belief that the only way to keep civilization functioning is by taking backups and testing them.

Recently I had few discussions with developers, and it turned out that ideas that I consider trivial can be viewed as deeply insightful by someone with different experience. Call that ideas arbitrage – goods that are common in one environment are valuable in another. Good hummus costs around 2$ in Tel-Aviv, while in Palo Alto I’ll be happy to pay many times that price if anyone would sell it!

I’m tired of spending time chasing pseudo-bugs that are actually problems in configuration. Things keep changing in QA, staging and production servers and I have to keep figuring out why my code no longer works

Figure out what your program needs in order to run properly. Memory, permissions, settings in files, etc. Then write a script that verifies all that.

This has 3 benefits:

  1. Sysadmins can check that the environment is correct before installing the software and before calling you for help.
  2. If something goes wrong, you can ask the sysadmin to run your script and send you the results – assisting in faster resolution.
  3. If the script doesn’t catch an issue and you have to spend hours debugging a broken configuration, you can later modify the script to catch the new issue and you’ll never have to spend those hours again.

I’m designing a new search system. Should I use Oracle or Voldermort?

Since you are asking me for advice, my guess is that neither of these solutions have a single compelling feature or limitation that make the decision clear-cut.
Therefore, go with the technology you understand better. Imagine yourself, a year in the future and the system just returned a wrong result. Which system you’ll find easier to solve the problem? Obviously the one you understand better, and the one that has troubleshooting tools you are more comfortable with.

Go with that one.


My users complain about a performance problem. My system is memory bounded and I believe that adding more memory will solve the issue. I opened a ticket for our sysadmins to add memory, but they have been ignoring my ticket and nothing was done. Now I’m blamed for not solving the performance issue!

Here are several possible solutions, ordered from recommended to highly risky:

  1. Update the ticket and ask for ETA.
  2. Use whatever internal process you have to escallate the ticket or make it higher priority.
  3. Make friends with a sysadmin and ask your friend to check the ticket and help you.
  4. Complain to your manager
  5. Complain to the manager of the sysadmin
  6. Go to the sysadmin cube and ask him nicely when the ticket will be ready
  7. Go to the sysadmin cube and yell
  8. Ask for a meeting or conference call involving more than 2 managers.

Interested in working with Gwen? Schedule a tech call.

3 Comments. Leave new

Problem: I want to write a module which will keep two databases refresh over the network, both synchronously and asynchronously. I need to confirm that your databases will not mind that.

Since the developer is asking from me, the DBA. My first reply is that its not just my database, it also belongs to you, and its due to the developers that database introduces more features and gets improved, because they always push the limits.

My second reply would be to not to reinvent the wheel and use the built in features of database to do that.

I have come across numerous such propositions from developers where they wanted to develop what was already there in “my” database.

Alex Gorbachev
August 12, 2010 1:51 pm

For the last problem, step zero is… BAAG. It’s likely not a memory problem and even if it is, you can likely fix it in the code. ;-)

Log Buffer #199, A Carnival of the Vanities for DBAs | The Pythian Blog
August 13, 2010 4:22 pm

[…] Kicking off this week are posts recommended by Gwen Shapira who also took a few minutes to share her production advice for developers. […]


Leave a Reply

Your email address will not be published. Required fields are marked *