Comments on: Source Controlling the Database Schema http://www.pythian.com/blogs/824/source-controlling-the-database-schema News and views from Pythian DBAs Sat, 22 Nov 2008 05:38:42 +0000 http://wordpress.org/?v=2.3.2 By: Fred Mitchell http://www.pythian.com/blogs/824/source-controlling-the-database-schema#comment-179381 Fred Mitchell Sun, 13 Apr 2008 09:50:48 +0000 http://www.pythian.com/blogs/824/source-controlling-the-database-schema#comment-179381 Sheeri, We are actually dealing with this very issue at OLB here. What we will probably do in the interim is to take a LVM snapshot of the entire MySQL database and "version control" that at the *release* level -- so when we do a *release* we'll be able to completely duplicate what went out to QA. Going from QA to Production is trickier, but what we are planning to do is create a cannned set of ALTER/GRANT statements to operate on the *prior* database snapshot, and this way we have complete reproducibility, at least in our dev/qa environments. There should NEVER be a need to backrev the production DB more than to the immediate prior release, and even then that should be a rare event. Sheeri, We are actually dealing with this very issue at OLB here. What we will probably do in the interim is to take a LVM snapshot of the entire MySQL database and “version control” that at the *release* level — so when we do a *release* we’ll be able to completely duplicate what went out to QA.

Going from QA to Production is trickier, but what we are planning to do is create a cannned set of ALTER/GRANT statements to operate on the *prior* database snapshot, and this way we have complete reproducibility, at least in our dev/qa environments. There should NEVER be a need to backrev the production DB more than to the immediate prior release, and even then that should be a rare event.

]]> By: Log Buffer #84: a Carnival of the Vanities for DBAs http://www.pythian.com/blogs/824/source-controlling-the-database-schema#comment-162215 Log Buffer #84: a Carnival of the Vanities for DBAs Fri, 15 Feb 2008 17:23:31 +0000 http://www.pythian.com/blogs/824/source-controlling-the-database-schema#comment-162215 [...] missed one post, though — Sheeri Cabral’s item on source controlling the database schema here on Pythian’s blog. Sheeri (a.k.a., the MySQL SheBA) also finally caves, and tells why [...] […] missed one post, though — Sheeri Cabral’s item on source controlling the database schema here on Pythian’s blog. Sheeri (a.k.a., the MySQL SheBA) also finally caves, and tells why […]

]]> By: Sheeri Cabral http://www.pythian.com/blogs/824/source-controlling-the-database-schema#comment-161795 Sheeri Cabral Thu, 14 Feb 2008 14:42:02 +0000 http://www.pythian.com/blogs/824/source-controlling-the-database-schema#comment-161795 Fred -- I think there's probably a limited dataset to which that applies -- for instance, if you have a list of "active countries" or something, or having a lookup table of static values, then yes, you want to keep that under revision control. The point is to say "go here, and give me a working application at that point." You don't necessarily need a lot of *user* supplied data, though you may need some static tables. If an application needs millions of rows to just function, it's probably a specialized application. Fred — I think there’s probably a limited dataset to which that applies — for instance, if you have a list of “active countries” or something, or having a lookup table of static values, then yes, you want to keep that under revision control.

The point is to say “go here, and give me a working application at that point.” You don’t necessarily need a lot of *user* supplied data, though you may need some static tables. If an application needs millions of rows to just function, it’s probably a specialized application.

]]> By: Heidi Schmidt http://www.pythian.com/blogs/824/source-controlling-the-database-schema#comment-161618 Heidi Schmidt Thu, 14 Feb 2008 02:58:03 +0000 http://www.pythian.com/blogs/824/source-controlling-the-database-schema#comment-161618 For tracking which statements were run when... I think more towards auditing (fine grained or basic or oracle vault) For auditing I've seen it is more important to audit the case not expected than to audit every possible instance. The volume of changes would dull the eyes looking for an exception to the pattern. It helps to have the business rules on hand to identify which is the exception and which is the rule. Source control is looking up though. In Oracle there are two ways you could go about doing Source Control of the schema ( and proabably a third if you sql traced the change management pack dbms_job session to see what package or SQL statements it ran and worked to take OEM Grid Control GUI out of the middle) Oracle Grid Control has built in the "change management pack" (been in some shape or form since 8i) which allows you to create a dbms_job that runs to capture the baseline of anything and everything (schema, user, tablespace, tables, views, etc) Then you can capture and publish that report to end users. It can publish in HTML or ASCII format. I've used it to hunt down the changes that occur from week to week. Or you can do wonders with command line control. In 10G using DBMS_METADATA and bit of tweaking in your SQLplus settings to spool the file out and automate a check in of the DDL. Then CVS or subversion diffs become the tool that shows what has changed. Consistently though, the capture has to be the same for both options. More so with version control software because the differentiation of what is a delta is done line by line and not at the higher object to object level the Change Management pack can yield. Heidi For tracking which statements were run when… I think more towards auditing (fine grained or basic or oracle vault)
For auditing I’ve seen it is more important to audit the case not expected than to audit every possible instance. The volume of changes would dull the eyes looking for an exception to the pattern.
It helps to have the business rules on hand to identify which is the exception and which is the rule.

Source control is looking up though. In Oracle there are two ways you could go about doing Source Control of the schema
( and proabably a third if you sql traced the change management pack dbms_job session to see what package or SQL statements it ran and worked to take OEM Grid Control GUI out of the middle)

Oracle Grid Control has built in the “change management pack” (been in some shape or form since 8i) which allows you to create a dbms_job that runs to capture the baseline of anything and everything (schema, user, tablespace, tables, views, etc)
Then you can capture and publish that report to end users.
It can publish in HTML or ASCII format. I’ve used it to hunt down the changes that occur from week to week.

Or you can do wonders with command line control. In 10G using DBMS_METADATA and bit of tweaking in your SQLplus settings to spool the file out and automate a check in of the DDL.
Then CVS or subversion diffs become the tool that shows what has changed.

Consistently though, the capture has to be the same for both options. More so with version control software because the differentiation of what is a delta is done line by line and not at the higher object to object level the Change Management pack can yield.

Heidi

]]> By: Fred Mitchell http://www.pythian.com/blogs/824/source-controlling-the-database-schema#comment-161514 Fred Mitchell Wed, 13 Feb 2008 21:49:32 +0000 http://www.pythian.com/blogs/824/source-controlling-the-database-schema#comment-161514 Source control on the database schema, while a laudable idea, is not without its problems. For starters, if you need to go to an earlier version of the schema, how do you do that? With the code, it's easy -- just recompile an earlier snapshot and you're done. But with a schema, there is also associated data, perhaps millions of rows, which would also have to be transformed. There is no straightforward means to go back to an earlier schema unless one wanted to forget about the data. And saving just the ALTER statements, for instance, does not allow one to incrementally go back to an older schema at all. So, you're kinda stuck. The only useful purpose of putting the schema under version control is to allow one to view the history only. Putting all the data as well under version control will be prohibitively expensive for anything but trivial datasets. Yet, not all is lost. Here lies possibilities for new and more powerful tools specifically tailored to handle the DBA's needs for keeping track of historical changes to the schema. Time for me to cogitate on this a bit. Source control on the database schema, while a laudable idea, is not without its problems. For starters, if you need to go to an earlier version of the schema, how do you do that? With the code, it’s easy — just recompile an earlier snapshot and you’re done. But with a schema, there is also associated data, perhaps millions of rows, which would also have to be transformed. There is no straightforward means to go back to an earlier schema unless one wanted to forget about the data. And saving just the ALTER statements, for instance, does not allow one to incrementally go back to an older schema at all.

So, you’re kinda stuck. The only useful purpose of putting the schema under version control is to allow one to view the history only. Putting all the data as well under version control will be prohibitively expensive for anything but trivial datasets.

Yet, not all is lost. Here lies possibilities for new and more powerful tools specifically tailored to handle the DBA’s needs for keeping track of historical changes to the schema. Time for me to cogitate on this a bit.

]]> By: Sheeri Cabral http://www.pythian.com/blogs/824/source-controlling-the-database-schema#comment-161402 Sheeri Cabral Wed, 13 Feb 2008 14:24:18 +0000 http://www.pythian.com/blogs/824/source-controlling-the-database-schema#comment-161402 Lukas, Actually GRANT and user management is, in my eyes, a simpler issue -- if a GRANT statement is issued twice, there isn't really a problem. Whereas if an ALTER TABLE statement is issued twice, you'll either get an error (ie, if you're adding a column) or a success but end up doing something bad like duplicating an index. :-\ Lukas,

Actually GRANT and user management is, in my eyes, a simpler issue — if a GRANT statement is issued twice, there isn’t really a problem. Whereas if an ALTER TABLE statement is issued twice, you’ll either get an error (ie, if you’re adding a column) or a success but end up doing something bad like duplicating an index. :-\

]]> By: Lukas http://www.pythian.com/blogs/824/source-controlling-the-database-schema#comment-161360 Lukas Wed, 13 Feb 2008 12:12:02 +0000 http://www.pythian.com/blogs/824/source-controlling-the-database-schema#comment-161360 This is indeed a huge issue that most people try hard to ignore and simply accept the consequences. Its been accepted as a fact of life by most programmers/DBAs it seems. I have put a fair bit of thinking on this topic, but for the most part I have only gotten to the point of documenting the different approaches and their pro's and con's. You can read somewhat recent posting on the topic here: http://pooteeweet.org/blog/918 Grant and user management stuff I have not really had any luck in putting on the map here at all yet. I guess you could come up with a system that makes it possible to parameterize the execution (or asks for any placeholders for which no values can be automatically determined). This is indeed a huge issue that most people try hard to ignore and simply accept the consequences. Its been accepted as a fact of life by most programmers/DBAs it seems.

I have put a fair bit of thinking on this topic, but for the most part I have only gotten to the point of documenting the different approaches and their pro’s and con’s. You can read somewhat recent posting on the topic here:
http://pooteeweet.org/blog/918

Grant and user management stuff I have not really had any luck in putting on the map here at all yet. I guess you could come up with a system that makes it possible to parameterize the execution (or asks for any placeholders for which no values can be automatically determined).

]]>