From the November 19th, 2008 press release:

The MySQL Query Analyzer saves time and effort in finding and fixing problem queries by providing:

• Aggregated view into query execution counts, run time, result sets across all MySQL servers with no dependence on MySQL logs or SHOW PROCESSLIST
• Sortable views by all monitored statistics
• Searchable and sortable queries by query type, content, server, database, date/time, interval range, and “when first seen”
• Historical and real-time analysis of all queries across all servers
• Drill downs into sampled query execution statistics, fully qualified with variable substitutions, and EXPLAIN results

Can you do all this with other tools or by hand?

Absolutely. Some of the those very tools I have discussed before. Most I have used at least once or twice. The question is really can Query Analyzer do this better? Find out the answer at the end!!

Installation

The Query Analyzer program is part of the Enterprise Monitor program. For those who are not familiar, Enterprise monitor is used for monitoring MySQL servers. Big surprise right? It works by having an “agent” installed on the server to be monitored which then passes information back to the central server which displays gathered information in an web-based interface.

I will confess. The installation of the service manager was performed by Sheeri. However, going through the notes she left it appears to have been rather straightforward. I picked it up where the configuration begins.

I will note one problem with configuration. Once the package is installed you have to do some setup before actually using the program. This is where it got a bit tricky. The test setup is on a system without direct Internet access (prevented by a firewall, for security reasons). The package has your key, your enterprise account name (email) and password. It connects to the Sun servers to verify your credentials and activate the program. Well, not if your server doesn’t have a direct connection to the Internet!

In that case you use a product key. Well, being new to this I had no idea where to get that from. I searched around a bit on the enterprise website and did not find it.  On the enterprise website I opened a ticket with Sun with a level of S3 (meaning this is non-production impacting..testing setup). I had a correct reply in FOUR minutes.

Sun support response time is very good! On the downside, this information probably should have been in the knowledge base.

This actually finished up the configuration of the Service Manager (the core functionality). Now to install the Service Agent which goes on the server(s) you are going to monitor. In this case the Agent was also being installed on the test system. As with the Service Manager the actual install was straightforward.

The program started up correctly the first time I tried. The proxy by default starts on port 4040 and I kept this as the setting.

Running the Program

Once this little quirk was resolved I began running some queries and seeing them show up in the Query Analyzer.

First impressions are that Query Analyzer does exactly what Sun says that it does. As you queries pass through the proxy and on to the database they are also sent to the Service Manager program. It slices,dices and display the data gathered in close to real time on the Query Analyzer tab of the dashboard.

Here are the various ways you can sort your queries:

• By Query
• By Database
• By total execution count
• By Execution time
  • total query execution time
  • maximum query execution time
  • average query execution time
• By Rows
  • total rows
  • maximum
• By Bytes
  • total bytes
  • maximum bytes
  • average bytes
• by date/time query was first seen

One click is all it takes to change the sorting method.

The Query Analyzer table can let you see at a glance what is going on with your system. Here is a a screen shot of the Query Analyzer (click to enlarge):

By default the Query Analyzer displays data for up to 24 hours. This can be altered by pulling down the “Time Display” option in the filter bar and choosing the “From / To” option instead of “Interval”. This will change the filter bar to have date based pick lists so you can specify a longer date range. Be warned - with a significant amount of data this can impact performance.

Any query can be drilled down into and examined, even showing the EXPLAIN of the query  if it took longer than 500 ms. Utilizing this you can begin your resolution of the problem in the typical MySQL manner.

The Final Analysis

I am not aware of anything else available either commercially or freely available that does this type of analysis for MySQL. The only similar functionality done by other products is through reading of the slow query log. While this can provide some of the information, the Query Analyzer does a better job at real-time analysis of the collected data with your sorting options.

While I am not familiar with other vendor’s database servers, I work with people who are. So I asked them. From what I understand there is somewhat similar functionality to the various programs that work with the slow query log, but nothing that is quite like the real-time analysis that Query Analyzer proves. If I am wrong, kindly suggest similar tools!

While the use of the proxy program provides added complexity, it also brings flexibility to the program.

Does the Query Analyzer do what it says it does? Yes! The only significant downside to this program is the cost. However, if you have Enterprise support it’s already included. So that’s not bad. I would not recommend that you buy Enterprise support JUST to get Query Analyzer — maybe Sun should package the Query Analyzer as a separate product? Reasonably priced (say, $200 or less) I would be the first person to tell you to buy it for any company with production systems.

As I said at the start, there other programs out there that can do grunt work that Query Analyzer does and help you resolve issues with slow queries. They don’t do this in real time and in addition the ease of use after configuration is what sets this apart. The simple graphical interface makes it dead simple  for everyone to see what is causing traffic to snarl. It would be nice if it resolved the problems for you…but I don’t know of any software that does that!!

I am losing count of how many of these Log Buffers I have done. If I recall correctly, this is number three. But then again, it could be number four. You know what they say though—the more the merrier!

This week’s Buffer features a great allegory about a database, some tips, feature requests, and breaking news. It seems that this week, people shared information that was relevant across database servers, so I recommend that no matter the database server you work on, you should read through each section today.

MySQL

Since the writer of the Buffer gets to choose the order of the stories, I will begin with my favorite db—MySQL. This week the MySQL world saw  some benchmarking of testing of recent improvements to the XtraDB storage engine. the discussion generated around what was done to improve the performance on a 16-core box was really interesting. I’m looking forward to working with the XtraDB engine.

And while only somewhat MySQL-specific, for those who have an interest in database warehousing, comes this interesting post from Alex Tomic on agile development with data warehousing. Bet you never though you would hear those two phrases together!

Gregory Haas talks about his methods for source control for his database servers. It’s something that many DBAs don’t consider. I won’t embarrass everyone  by asking for a show of hands of who has their schema and tools under source control.

Now that MySQL server 5.1 has been GA for a while, it is starting to roll out into production at some sites. Karlsson writes about his use of events.  I will leave it to the gentle reader to determine the very important benefit he points out about events. Thanks for the tip Karlsson!

For those still using 5.0 ( . . . um . . . the approximate 99.9% of you), Yves Trudeau Matt Yonkovit shows how with a little manual hackery you can gather the queries that are hitting your server”at will”. While not as elegant as 5.1’s dynamic logging, it certainly does the job. Thanks Matt!

Baron has a good idea: a reverse changelog.  If you have ever worked your way through a changelog in preparation for an upgrade, this will make sense to you.

My colleague Sheeri wrote a great blog post on MySQL’s default silent truncation behavior and how this can affect your database server’s data integrity.

Mark Callaghan, Google wizard of the dolphin, writes about global mutexes and how they are limiting the SMP performance  of MySQL server. There has been much discussion about this over the last year and it seems some progress is being made. Brian Aker writes down his thoughts on the matter as well.

Finally, for those going to the MySQL User’s Conference, Giuseppe Maxia officially announced the MySQL Camp, which will run in conjunction with the User’s Conference.

Microsoft SQL Server

You know there are people who think DBAs are boring and stuffy. I think these people should all meet Simon Sabin. This chap from the UK has a twisted sense of humor, and he manages to relate a fish and chip shop to a database server. Take time to read through. It is absolutely hysterical!

On a more serious note. What does it take to build the 60GB index for a billion-and-a-half rows? Well the SQL Fool knows. And she shows you how to execute a well-crafted plan to do this without bring the database to its knees (and keeping your users from being howling mad).

Buck Woody points to a site with an SQL formatter available online. Not only does it format SQL for Microsoft SQL server, it also does Oracle, MySQL, and DB2. Not only that, but it will translate the output to a multitude of programming languages including C#, Java, and PHP to name a few. It looks a really cool tool.

Speaking of tools, Kalen Delaney provides a nice list of free troubleshooting tools available for SQL Server.

Need a random number? George Mastros give some great tips on generating numbers for creating tables for testing.

Are you someone who has to hire a senior DBA? Brent Ozar has a list of 10 questions you can ask in the interview process. Good questions, Brent!

Oracle

I’ll kick this off with some basic material excellently presented. Eddie Awad points to a 9.5 minute video on indexing by Stephane Faroult. Stephane is the the author of The Art of SQL (a book I highly recommend). Arup Nanda has a very good post about how to alter a materialized view of any size in seconds.

Lewis Cunningham’s blog is always a good read. It seems he just got involved with agile development. He gives his perspective on the process.

Tanel Poder gets in-depth with cursors in Oracle. He calls it Multipart cursor subexection and PRECOMPUTE_SUBQUERY hint.

Want to learn about the ALL_TABS_COLUMN table? Don’t know what it is? Join Robert Vollman—AKA the Oracle guru—as he schools a pupil in the how you can use the ALL_TABS _COLUMN to find out information about your tables columns.

Joel Goodman talks about different ways an ASM external storage solution can be integrated with your Oracle RAC or single-instance server.

Chen Shapira, the Simple DBA, offered some good theory and practical application on queues.

PostgreSQL

In the world of PostgreSQL,  Gavin brings us some benchmarking results comparing two GNU/Linux filesystems, ext4 and ext3. If GNU/Linux is running on your servers, you need to take a look at this post.

That is all for this edition. I hope you had as much fun reading it as I had writing it. If I missed you favorite DB blog from this week, please leave a comment with a link. Also, I encourage you to give it a shot as well—drop Dave a line and get Log Buffer going on your own blog.

See you all soon!

1. There are numerous calculations available online.
2. It seems that none of them take everything in account.

Wouldn’t it be nice if there was a total possible memory consumption status value? It can be a valuable piece of information. (And while I am requesting things, what about peak memory usage by the mysqld server since it has been online?) Realistically, there is almost no case where you would actually use the maximum amount of memory. But knowing how much memory that MySQL server could possibly use would be a very useful bit of information to know.

Second, why isn’t there an “official” formula that actually shows everything? It’s easy to forget things. Just as the simple case, what about the query cache? Did you remember it?

Sure, there are some big things. innodb_buffer_pool_size and key_buffer_size spring to mind. But all the little things add up too. On a busy system did you consider thread_stack? It defaults to 128k per connection. With 500 connections that’s 64 MB. You may dismiss that and and say its not much. But in the end it doesn’t take 64 MB to cause a crash of your serve.

Do you know how much the core of MySQL server actually uses outside of all the buffers and data structures it builds? The only figure I have seen is 32 MB.

What about the operating system?  And I don’t even want to consider if other programs are running on the same server.

max_memory_needed = core_mysql + global_values + (thread_buffers * max_connections)

max_memory_needed = core_mysql
    + key_buffer_size
    + innodb_buffer_pool_size
    + innnodb_additional_memory_pool_size
    + innodb_log_buffer_size
    + max_tmp_tables * min(tmp_table_size,max_heap_table_size)
    + query_cache_size
    + 3 * myisam_sort_buffer_size
    + max_connections * ( read_buffer_size + join_buffer_size + read_rnd_buffer_size + thread_stack + (2 * max_packet_size) ))

core_mysql is the amount actually used by the core server itself without all the buffers and data structures that are given above.  The only figure I have seen for this is 32 MB which is probably pretty close.

The myisam_sort_buffer_size is the size of the buffer allocated when sorting MyISAM indexes during a REPAIR TABLE, or when creating indexes with CREATE TABLE or CREATE INDEX. I am estimating there will never be more than three of these operations going on at the same time. Of course there is no guarantee of this.

I believe the above formula will give a pretty accurate representation of the total amount of memory that could possibly be used by a MySQL server and operating system.

Where it becomes far more unclear is a more realistic “normal operating conditions” formula. The problem here is that this is going to vary from application to application. One application might never hit more than 75% of the memory calculated by this formula.  Another might be 50%, and another 87%.

How do you resolve this?  Well, probably the best way is through long-term monitoring. Cacti or Ganglia spring to mind for this. If you configure a server conservatively and know that it is using 80% of RAM at any time over a significant period of time (including periods of time with spikes in server activity), then you know you can increase the amount of memory allocated somewhere.

What I typically do (if I have a choice), is configure a server for the expected number of connections, and then, after establishing a baseline of a set amount of connections and a minimum amount of buffering, I monitor the server while it is in use. Then I add any additional memory the server can spare to the InnoDB buffer pool (this is presuming that InnoDB is the main storage engine in use). Always leave some free memory and don’t allocate everything down to the last byte. Once MySQL starts hitting your swap partition, very bad things can happen with servers slowing down and users complaining.

How do you determine your buffer allocations and memory usage? Is it just hit-or-miss? And did I miss something in the above formula? I want to hear from you!

You can reach me with your ideas at bmurphy@paragon-cs.com . I need to see all article proposals by the 15^th of this month. Draft articles must be in by the 10^th of January, so if I accept your proposal, you have a month to get things ready. I should also mention that in the past we have used blog postings (with author permission and full credit of course). If you have blogged something recently that you feel is worthy of publishing in the magazine, send me the link and I will take a look.

Never heard of MySQL Magazine? It is a digital magazine designed for both database administrators and developers of the MySQL database server. All six back-issues are available for free download from the website.

The database system must resolve the situation where, for whatever reason, the database server stops working in the middle of a transaction.  Perhaps the power goes out, the hard drive crashes, or the CPU disappears in a cloud of smoke. Maybe the MySQL Server itself crashes for some reason. What does the MySQL Server do when operation resumes?

Potentially, there are transactions in an inconsistent state. How are inconsistent transactions resolved? They are resolved through a recovery process involving log files that are called transactional logs. There are two kinds of transactional logs: redo logs and undo logs.

Redo logs are used to apply changes that were made in memory but not flushed to the permanent table records (for example, InnoDB’s permanent table records are the ibdata files.) This situation would typically happen after a MySQL Server crash.  Logging is done this way because it allows faster database operation.

This might be seem counter-intuitive at first look. Instead of writing to one file (the permanent table records file) you are writing to two different files (the redo log file and the permanent table records file). The difference is that the writes to the database table file(s) are random in nature while the writes to the redo log file will be sequential, which is usually going to be much faster. You perform the much-faster writes to the redo log as needed and then perform the slower writes to the table files periodically when there is time. Thus, the system actually operates faster writing to both files rather than only one file. This applying of the redo logs occurs automatically on startup of the MySQL server after a crash. Very similar processes are used by other database platforms.

In addition to the redo log there must also be undo logs. When a database user starts a transaction and executes some commands, the database does not know if the user will end the transaction with a COMMIT or a ROLLBACK command. Ending with  a COMMIT means all the changes made in the course of the transaction have to be preserved (fulfilling the Durable aspect of ACID). If the transaction gets interrupted for some reason, such as the MySQL daemon crashing, the client disconnecting before sending a COMMIT, or the user issuing a ROLLBACK command, then all changes made by the transaction need to be undone.

If the server crashed, the redo log files are applied first, on start up. This puts the database in a consistent state. Now the database server will need to roll back the transactions which were not committed but had  already made changes to the database. Undo logs are used for this.  As an example, if you are running a transaction that adds a million rows and the server crashes after eight hundred thousand inserts are performed, the server will first use the redo log to get the database server into a consistent stand than then will perform a rollback of the eight hundred thousand inserts using the undo logs. For InnoDB, this undo information is stored in the ibdata file(s). For Falcon the undo log information is stored in the serial log file.

Transactional logs have some common characteristics. The log records each contain Log Sequence Numbers (LSN) which are unique IDs for each record. There is a also a Previous Log Sequence Number (Prev LSN), which is a link to the last log record. A Transaction ID number is used to reference the database transaction generating the log record.  Finally, checkpoints are entries in the transactional logs that are used to speed up the processing of those logs when they are needed. Checkpoints contain a list of transactions that are open when the checkpoint was created. This is used by the server to determine how far back into the log it must go to begin processing.

(A small side note. Some people get confused thinking the binary log files are used in this process. That is not true. The binary logs are used only for replication purposes and for manual data recover situations, such as when a backup is restored and you need to “catch the server up” using the binary logs. The server does not use these files in the transactional recovery process.)

That completes this general overview of the transactional logs in MySQL (or any modern RDBMS really). I hope you find it useful.

While I could spend an entire blog post—if not several—on a definition of cloud computing, I will be talking only about cloud computing in the sense of companies moving servers from their building or network operations center to running virtual servers in this computing cloud.

While there are a number of companies providing virtual servers, the most visible is Amazon, with their Amazon Web Services (AWS). I will be talking about AWS in this post as it is the service with which I am most familiar. It seems like every month, AWS rolls out new options and services. Just recently Amazon announced that you can now run on AWS the Windows operating system along with SQL Server.

Amazon also announced a service level agreement (SLA) of 99.5%. The SLA is important. It is a guarantee of service uptime. If Amazon don’t meet the SLA, then you get money back. As any of you will know, you have to be able to count on your data center. 99.5% is a pretty good level of coverage.

Beyond that, one of the new features Amazon will be implementing during the next year is the use of regions and availability zones. Regions are distinctly different areas of a country (or completely separate countries); availability zones are designed to be insulated from failures in other availability zones and provide inexpensive, low-latency network connectivity to other availability zones in the same region. What does this mean? It will soon be very easy to deploy a set of servers in different areas and/or regions so that your data and servers are spread out and not vulnerable to a single point of failure.

I am not going to go into any detail about how virtual servers work. That’s not the point of this post. I am going to concentrate on what you can do with virtual servers.

Once you are properly set up, you can bring new servers online in just a few minutes. Got a spike in your app usage due to Digg coverage or a big sale? Bring X number of servers online to handle the load, and then after it passes, shut down the now-unneeded servers. Simple, cheap and fast.

In a traditional setting where the company maintains its own servers, they either have to order servers far in advance and having them sitting idle in the racks at the data center. Or risk not having enough compute power in place. If this taken into account, the savings offered a company by a cloud-based approach can really begin to add up.

A simple way to begin using the cloud would be to use the S3 storage service to store your backups from your current databases. In one simple step you have off-site storage of unlimited capacity, available anytime you need it. Use the Amazon Simple Queuing System (SQS) to build a network of servers that can operate on a queue of tasks in parallel. While you could develop this setup internally, it would take a lot of time, energy, and money. With SQS, you can bypass all that and be running in days, if not hours.

These are just the obvious things that spring to mind. I am sure over the next few years we will see many innovative applications and uses that I can’t imagine right now.

Are there limitations? Of course. Currently, RAM is limited to 15 gigs with four cores of CPU power. Amazon offers a SAN-based permanent storage called Elastic Block Storage (EBS). I have seen no data regarding the actual throughput of EBS, so I don’t know how suitable it would be for a very high-end system. For many systems, however, this would be perfectly adequate.

Overall, I think the flexibility that a system such as AWS provides will allow a new generation of companies who are quicker to market, more agile to make changes and have lower overhead. All good things in this economy!

I am certain that at the MySQL Users Conference next year, there will be many talks on how to leverage cloud computing with MySQL. I look forward to that.

Thanks,
Keith.

Beginning with the world of MySQL.

Monty Taylor kicks things off, bringing us news of the ability to use the innodb plugin with Drizzle. I just think its great how the Drizzle development is moving along so rapidly. My thanks to everyone involved.  Monty also puts out a call to help defray the cost of the purchase of the drizzle.org domain name.

There have been several posts this week on virtualization of MySQL Server. I point you to a colleague at Pythian, Sheeri Cabral, who wrote about some of the benefits our customers have already experienced while using virtualized servers. While virtualized servers are not a panacea, they definitely have a place in the DBA’s environment. If you don’t work with any currently, you will be doing so down the road.

The OpenSQL “unconference” is coming! The date is Nov 14 -16 in Charlottesville, Virginia, USA. Details and registration information are available here. Speaking of conferences, the MySQL Users Conference, which is in April in Santa Clara, CA, USA has extended its deadline for topic proposals.

Baron has a great post on naming conventions for your schema.

In the how-to department, Falko Timme has a good tutorial on how to set up GreenSQL to protect your databases from SQL Injection attacks. Finally, Johan Andersson has a good introductory blog post on optimizing queries for a NDB cluster, and a post on how you can get the Cluster Sandbox tool for testing the MySQL Cluster.

The Oracle world had sad news this week. Carl Backstrom, who was an Oracle employee and APEX developer, was killed early Sunday morning in a car accident in Nevada. Many people have written their thoughts about this, and I give you the links in no particular order:

http://joelkallman.blogspot.com/2008/10/carl-backstrom.html
http://tonguc.wordpress.com/2008/10/28/we-lost-carl/
http://www.liberidu.com/blog/?p=576
http://fuadarshad.blogspot.com/2008/10/rip-carlback.html
http://www.oraclenerd.com/2008/10/carl-backstrom.html
http://spendolini.blogspot.com/2008/10/carl-backstrom-tragic-loss.html

This week brings us the introduction of a new beta version (4.0 beta 7) of TVD$XTAT, Christian Antognini’s freeware  tool for working with raw SQL trace files.

Even though the Exadata server was introduced some time ago it is still generating comment in the Oracle users community. Greg Rahn from the Oracle Real-World Performance Group at Oracle had some thoughts about some recent comparisons between the Exadata servers and EMS  arrays.

An interesting observation comes from Robert Hodges about building highly reliable database systems (not just for Oracle!).

Marc Billette brings some interesting information about how you can monitor the amount of time it takes Oracle to apply logs. Jason Arneil has some very noteworthy work showing how performing an online rebalance of ASM can affect your system with I/O saturation. Richard Foote has some very in-depth details for those who want to know what is going on with indexes and how indexes can become fragmented even when deletions of the index are not occurring.

Looking to automate your Oracle installations more? Take a look at Jeremy’s idea for a project to automate the install patch process.

The world of SQL Server brings us a lot of news this week.

Tony Rogerson shows a simple method of creating an HTML table with table data from query results. All done in SQL! Another nifty coding idea to add to your arsenal is given by the SSIS Junkie where he talks about nesting variables to calculate values.

SQLServerpedia reports that Windows Server 2008 will only be offered in a  64-bit version. This should simplify the life of some SQL Server DBAs! New functionality for SQL Server 2008 was written about by Paul Randal with the re(addition) of minimally-logged inserts for bulk data loads. And finally, and maybe most important, Paul also blogs about something near and dear to my heart: backups and the recovery process.

Anyone who quotes Eric Raymond immediately gets high marks in my book. Ken Downs, The Database Programmer, does just that in his great post about the assertion that people make (erroneously, I believe) that code is data. Ken, your post made my week!

Continuing the coding examples, the Patron Saint of Lost Yaks writes about using bayesian estimates for averages.

PostgreSQL

The Postgres online journal was busy this week. They have a good post on why sometimes indexes aren’t used. Also, the journal has a good second part of an article on how to write Postgres functions.

Selena writes a nice summary of happenings at the Google Summer of Code Mentor Summit. And a trick or treat from Depesz who bring us a trick on how to set a field based on the sort order.

Informix

For those working with Informix, Guy Bowerman has written a good book review of Administering Informix Dynamic Server.

DB2

The DB2 DBAs had a conference this week—the IOD (Information on Demand) conference. Several people have blogged about it. Willie Favero shared his thoughts on IOD here.

Here’s what Beth Breidenbach had to say about Monday at IOD, and Adam Gartenberg wrote about Tuesday at the conference.

That’s all. Thanks for your attention, and I’ll see you again soon.

Recently I was working on a problem for a customer. They are converting a server with two InnoDB data files to innodb_file_per_table:

$ ls -l ibdata*
-rwxr-xr-x   1 mysql    mysql       2.0G Oct 14 13:10 ibdata1
-rwxr-xr-x   1 mysql    mysql       350G Oct 14 13:10 ibdata2

I honestly don’t recall ever seeing (or hearing about) so large a data file.

The method chosen for conversion boils down to this:

1. stop and start the server to enable innodb_file_per_table
2. alter all tables to myisam
3. stop server and delete ibdata file
4. restart server
5. convert tables back to InnoDB
6. add foreign keys

This post isn’t about how we did it, or about whether or not it was the best way, or anything like that. This post is really about the the last two steps.

There were over 500 InnoDB tables and something like 375 foreign keys. So I needed to somehow create the ALTER TABLE statements for these 500 servers. To drop the Foreign Keys, I used the following SELECT statement:

SELECT CONCAT( "mysql -u root -pPASS ", table_schema," -e 'ALTER TABLE ",table_name, ' DROP FOREIGN KEY ', constraint_name,';'' &') AS list_of_fks FROM information_schema.KEY_COLUMN_USAGE WHERE REFERENCED_TABLE_NAME is not null ORDER BY TABLE_NAME, COLUMN_NAME;

This created a list of statements that I could then put in a batch file, edit a little, and have them run in parallel.

I used the following SELECT statement to find all InnoDB tables and prepare a command to convert them to MyISAM:

SELECT CONCAT("mysql -u root -pPASS ", TABLE_SCHEMA, " -e 'ALTER TABLE ", TABLE_NAME, " ENGINE=MYISAM;' & ") FROM INFORMATION_SCHEMA.TABLES WHERE ENGINE='INNODB';

Again, the output can be put in a batch file, modified a little, and run in parallel.

Once everything was converted to MyISAM, I shut down the database and removed the InnoDB log and data files and modified the my.cnf to so I had file_per_table and a much smaller InnoDB data file. That was the fairly straightforward part. Now the fun began. I needed to convert all the tables back to InnoDB and add the FKs back.

Now ordinarily, I’m a command-line kind of of guy. However, having used diff before for comparisons, I decided to go another route. Based on some previous experience, I chose SQLyog for the job. Before I began any conversion of the tables I dumped the schema of the database and loaded it into a MySQL server running on my laptop.

Once all the conversions were done I used a tool in SQLyog called a “Schema Synchronization Tool”.  Essentially, you point the tool a “source” database and a “target” database. (It is very important to keep track of which is which.) During the choice of the source and target you also choose the database that you want to compare. Once this is done you simply click compare and it checks both database and then returns a nice list of SQL statements.

Since we had 500 tables to convert I didn’t just copy the SQL into a file and import it. I created more files that contained lines similar to:

mysql -u root -pXXX "ALTER TABLE .... " &

This way, they ran in the background. At one point I had over 200 alter statements running at the same time.

My one complaint about SQLyog is that the SQL it generated was sometimes inefficient. It would give me a SQL statement to ALTER TABLE engine=Innodb and then three separate statements for three different FKs. Each time it would have to copy the table so this would take four times as long as combining all these statement into one.  It is perfectly feasible to do this:

ALTER TABLE X ADD FK X1, ADD FK X2, ADD FKX3, engine=innodb;

Big time savings!

Unfortunately, I did run across what amounts to a bug. Not in SQLyog, but in MySQL. You can run this:

ALTER TABLE X ADD FK X1, ADD FK X2, ADD FKX3;

and MySQL will happily take the statement and run. Hours later when it gets done, you find out that the table is still MyISAM and the FKs were just turned into indexes. Stupid!

Sheeri pointed out that the MySQL Workbench program does something very similar. I haven’t had a chance to look into it yet. Having worked with MySQL Workbench before, I am sure it does a schema sync very well. Either way, trust me on this—if you have a significant schema synchronization project, you need to look into one of these tools.

I am not going to turn this into a rant about who is right, who is wrong, or  what should be done about it to resolve the problem. This isn’t the place. I probably don’t even have the right answer. I have a different angle.

If you are involved with MySQL as a database administrator, or if you work directly with MySQL in some other aspect, you can probably breathe a little easier. Why is this? MySQL Server has grown in market penetration for a long time. It is now a significant section of the RDBMS pie.  I predict that this market penetration will only continue to grow. As this economic downturn/recession/whatever continues, companies will look harder for ways to save money. What better way to do so than replace your proprietary RDBMS that can cost you significant amounts of money, with MySQL Server? For all intents, the same functionality is there, the speed and flexibility is certainly there, and there is a giant company behind MySQL now, providing “enterprise-ready” support.

The market is crying right now for MySQL database administrators. We don’t cost any more than Oracle or Microsoft DBAs, you know. Just a couple of years ago, very few companies hired MySQL DBAs. They hired developers who also did database administration, or a system administrators who also managed the MySQL server. Now, as the number of database servers increases and the amount of data grows they want real, honest-to-goodness database administrators. If you have production experience with MySQL server in any significant amount you will not have any problems finding a job. I don’t think this is going to change anytime soon. So, even if your company succumbs to the times, there are others out there who need your experience. Don’t be dismayed!

And companies who might be reading this—be smart. I know it costs time and dollars to move your infrastructure, but this is time and money well spent. Larry and Bill both have plenty of money and they really don’t need you or your company to continue to line their pockets with upgrades and maintenance and whatever. There is very little that either of their servers can do that MySQL can’t do right now. Why are you waiting?

MySQL too small for you? Well, Sun owns MySQL now. Ever heard of them? If you are so enterprise-uptight that you want one vendor to provide support for everything, congratulations. You should be in heaven. Sun does servers, Solaris for the OS, and of course, MySQL. Need middle-ware? Ever heard of Java? Might not be what I would recommend, but if it floats your boat, Sun can certainly provide all this and support too.

One thing I have found is that MySQL DBAs love to work remotely. In the job of a DBA it is often perfectly reasonable to work off-site. It requires a little “out-of-the-box” thinking, but the rewards are worth it. If you want the best for your company, you might want to consider this as an option.

If you are a working MySQL DBA and  are worried about the economy, you can worry a little less. While your house might be losing value, and your 401k has fallen on it’s face, you have a job. And there is a good chance it (or another one) will be there even when Wall Street is singing the blues.