The unsung heroes of InnoDB are the logfiles. They are what makes InnoDB automatic crash recovery possible.

Database administrators of other DBMS may be familiar with the concept of a “redo” log. When data is changed, affected data pages are changed in the innodb_buffer_pool. Then, the change is written to the redo log, which in MySQL is the InnoDB logfile (ib_logfile0 and ib_logfile1). The pages are marked as “dirty”, and eventually get flushed and written to disk.

If MySQL crashes, there may be data that is changed that has not been written to disk. Those data pages were marked as “dirty” in the innodb_buffer_pool, but after a MySQL crash the innodb_buffer_pool no longer exists. However, they were written to the redo log. On crash recovery, MySQL can read the redo log (InnoDB log files) and apply any changes that were not written to disk.

That is the basic functionality of the InnoDB log files. Given this, let’s look at some of the different parameters and their ramifications.

innodb_log_files_in_group is set with a default of 2. The logfiles are written in a circular manner — ib_logfile0 is written first, and when it has reached its maximum size, then ib_logfile1 will be written to.

innodb_log_file_size is the size of each log file in the log group. The total, combined size of all the log files has to be less than 4 Gb (according to the MySQL manual). Because the logfiles contain changes in the buffer pool that have not been written to disk, the total, combined size of all the log files should not be more than the innodb_buffer_pool_size.

If all the log files in the group are full of changes that have not been written to disk, MySQL will start to flush dirty pages from the InnoDB buffer pool, writing the changes to disk. If the log files are small, changes will be written to disk more often, which can cause more disk I/O.

When InnoDB does a crash recovery, it reads the log files. If the log files are large, it will take longer to recover from a crash. If innodb_fast_shutdown is set to 0, the log files are purged when MySQL shuts down — larger files mean a longer shutdown time. The default for innodb_fast_shutdown is 1, which means that the log files are not purged before a shutdown. Starting in MySQL 5.0.5, you can set it to 2, which simulates a crash, and at the next startup InnoDB will do a crash recovery.

innodb_flush_log_at_trx_commit controls how often the log files are written to. A value of 0 causes the log files to be written and flushed to disk once per second. The default is 1, which causes the log buffer to be written and flushed to disk after every transaction commit. The value can also be set to 2, which causes the log buffer to be written after every transaction commit and flushes the log files to disk once per second. A value of 2 means that MySQL might think that some changes are written to the log file, but do not persist in the log file after an operating system crash, because the log file was not flushed to disk before a crash.

Note that some filesystems are not honest about flushing to disk, so even though you may have the default value of 1, your system may be acting as if it has a value of 2. Setting this parameter to 2 means that there will be less I/O, at the cost of not being able to recover data from a crash.

innodb_flush_method changes how InnoDB opens and flushes data and log files. See the manual for details; the end result is a tradeoff in I/O performance versus whether or not an operating system crash would leave the InnoDB log files in an inconsistent state.

innodb_log_buffer_size is the write buffer for InnoDB log files. The larger the buffer is, the less often the log files are written to. This can save I/O.

As I putter around the MySQL INFORMATION_SCHEMA, I am finding lots of undocumented behavior for fields that should be straightforward. For example, the VIEWS table holds information about views, and the VIEW_DEFINITION field contains the view definition, right?

Well, when I was looking at the VIEW_DEFINITION today, I noticed an odd thing. Even though I had permissions to see the view definition (as proven by the SHOW CREATE VIEW command), the INFORMATION_SCHEMA.VIEWS table sometimes came up blank for the VIEW_DEFINITION. I had to figure out why, and now that I know, I’m not sure if it’s a bug or a feature…..can you figure it out?

mysql> USE INFORMATION_SCHEMA;
Database changed
mysql> SELECT TABLE_NAME,VIEW_DEFINITION FROM VIEWS WHERE TABLE_SCHEMA='sakila';

+----------------------------+-----------------+
| TABLE_NAME                 | VIEW_DEFINITION |
+----------------------------+-----------------+
| actor_info                 |                 |
| customer_list              |                 |
| film_list                  |                 |
| nicer_but_slower_film_list |                 |
| sales_by_film_category     |                 |
| sales_by_store             |                 |
| staff_list                 |                 |
+----------------------------+-----------------+
7 rows in set (0.16 sec)

mysql> SHOW CREATE VIEW sakila.actor_info\G
*************************** 1. row ***************************
                View: actor_info
         Create View: CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL
SECURITY INVOKER VIEW `sakila`.`actor_info` AS select `a`.`actor_id` AS `actor_i
d`,`a`.`first_name` AS `first_name`,`a`.`last_name` AS `last_name`,group_concat(
distinct concat(`c`.`name`,': ',(select group_concat(`f`.`title` order by `f`.`t
itle` ASC separator ', ') AS `GROUP_CONCAT(f.title ORDER BY f.title SEPARATOR ',
 ')` from ((`sakila`.`film` `f` join `sakila`.`film_category` `fc` on((`f`.`film
_id` = `fc`.`film_id`))) join `sakila`.`film_actor` `fa` on((`f`.`film_id` = `fa
`.`film_id`))) where ((`fc`.`category_id` = `c`.`category_id`) and (`fa`.`actor_
id` = `a`.`actor_id`)))) order by `c`.`name` ASC separator '; ') AS `film_info`
from (((`sakila`.`actor` `a` left join `sakila`.`film_actor` `fa` on((`a`.`actor
_id` = `fa`.`actor_id`))) left join `sakila`.`film_category` `fc` on((`fa`.`film
_id` = `fc`.`film_id`))) left join `sakila`.`category` `c` on((`fc`.`category_id
` = `c`.`category_id`))) group by `a`.`actor_id`,`a`.`first_name`,`a`.`last_name
`
character_set_client: latin1
collation_connection: latin1_swedish_ci
1 row in set (0.02 sec)

mysql> SHOW CREATE VIEW sakila.customer_list\G
*************************** 1. row ***************************
                View: customer_list
         Create View: CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL
SECURITY DEFINER VIEW `sakila`.`customer_list` AS select `cu`.`customer_id` AS `
ID`,concat(`cu`.`first_name`,_utf8' ',`cu`.`last_name`) AS `name`,`a`.`address`
AS `address`,`a`.`postal_code` AS `zip code`,`a`.`phone` AS `phone`,`sakila`.`ci
ty`.`city` AS `city`,`sakila`.`country`.`country` AS `country`,if(`cu`.`active`,
_utf8'active',_utf8'') AS `notes`,`cu`.`store_id` AS `SID` from (((`sakila`.`cus
tomer` `cu` join `sakila`.`address` `a` on((`cu`.`address_id` = `a`.`address_id`
))) join `sakila`.`city` on((`a`.`city_id` = `sakila`.`city`.`city_id`))) join `
sakila`.`country` on((`sakila`.`city`.`country_id` = `sakila`.`country`.`country
_id`)))
character_set_client: latin1
collation_connection: latin1_swedish_ci
1 row in set (0.00 sec)

mysql>

(some people read the dictionary, I read the data dictionary!)

A while ago, MySQL developed a Community Contribution Agreement for community contributions to the MySQL source code. While browsing the MySQL Forge Wiki I came across:

http://forge.mysql.com/wiki/Community_Contributions

This page shows that the Community Contribution Agreement has changed — it is no longer the document MySQL AB created. It is now Sun Microsystem’s standard Sun Contributor Agreement, which CEO Mårten Mickos recently explained to me was “more accepted than the agreement MySQL had come up with.”

I am happy to see some of the great Sun practices trickle down to replace some issues that MySQL did not handle smoothly. All in all, I agree with Mårten Mickos and think the Sun Contributor Agreement is much better….

….but what do you think?

So you want to store URLs in MySQL, and the URLs have those annoying %20%27%7C%26%5E%2B%2D%25 symbols? And you want to be able to show your users some kind of human-readable information. You might want to consider using this trick. Take this list of commonly escaped characters as an example:

%20 - space
%27 - '
%7C - |
%26 - &
%5E - ^
%2B - +
%2D - -
%25 - %

So, how about we do some search’n'replace on that?

mysql> SET @url:='%20%27%7C%26%5E%2B%2D%25';
Query OK, 0 rows affected (0.00 sec)

mysql> SELECT @url as original,
    ->        REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(
    ->        REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(
    ->        @test,'%20',' '),
    ->        '%27','\''),
    ->        '%7C','|'),  -- REPLACE() is case sensitive
    ->        '%7c','|'),  -- so we have
    ->        '%26','&'),
    ->        '%5E','^'),
    ->        '%5e','^'),  -- to repeat
    ->        '%2D','-'),
    ->        '%2d','-'),  -- ourselves
    ->        '%2B','+'),
    ->        '%2b','+'),  -- sometimes
    ->        '%25','%') as replaced;
+--------------------------+----------+
| original                 | replaced |
+--------------------------+----------+
| %20%27%7C%26%5E%2B%2D%25 |  '|&^+-% |
+--------------------------+----------+
1 row in set (0.01 sec)

mysql>

We can easily turn this into a function: (more…)

Welcome to the 116^th edition of Log Buffer, the weekly review of database blogs.

This was the week of Oracle Open World (OOW), Oracle’s gigantic annual get-together in San Francisco — always the heaviest week in Oracle blogs, so let’s start there.

For day-by-day coverage of OOW on the ground, I recommend Doug’s Oracle Blog: OOW Day 1, OOW Day 1.5, OOW Day 2, OOW Day 3.

Tom Kyte shared a podcast from OOW 2008, and interview with Oracle Magazine editor Tom Haunert, in which Tom, “ . . . stirs things up in this conversation about Oracle OpenWorld happenings, a new approach to publishing, and the trouble with triggers.”

Oracle teased everyone right at the beginning with word that CEO Larry Ellison’s keynote, carrying the title “Extreme Performance,” would introduce something big and new. And there was much speculation in the blogging world, some of it quite perspicacious. “Big and new” was soon going by the tantalizing nom-de-hype “X”. And before Larry’s keynote was even over (before he mothballed the black mock-turtleneck for another year), X was no longer unknown.

Writes Lucas Jellema on the AMIS Technology blog — The secret is out: Oracle launches “The Database Machine” - becoming a hardware vendor! “The big announcement that had loomed over the conference has been made. Oracle - in joint partnership with HP - introduces the world’s fastest hardware for running databases and especially data warehouses: the Exadata Storage Server.” Click through for Lucas’s précis of what it’s all about.

On blogs.oracle.com, Jack Flynn has some video excerpted from the keynote.

Lucas’s story has a picture of the thing itself, albeit a somewhat blurry one. Here’s a better image of one of the two new machines, the Exadata. Oooh, just look at it! Cor!

(more…)

Over the last few weeks I’ve been looking at several customers’ slow query logs, and I found in many of them an odd type of query. These are SELECT statements that contain an IN clause that includes dozens, sometimes hundreds of values. These statements often end in the slow query log. I’m not sure if these queries are this way by design or if they are generated by a specific database development tool.

I did some tests in one of my own databases, one with only around 10K rows in its largest table. The database corresponds to the Amarok media player. For example, I queried for songs by B. B. King (spelled “BB King”, “B.B. King”, etc. or with other artists: “B. B. King & Eric Clapton”).

The first query used a JOIN and an IN clause with all the spellings in my db; the second used the same JOIN and WHERE ... name LIKE "BB%" OR name LIKE "B.%". Both had the same execution plan, and both retrieved the same number of results. In MySQL version 4.1 there were some enhancements to the optimizer for treating these massive IN clauses, which means that for smaller databases, this is expected.

With bigger databases and more complex queries, things are different. (more…)

If you track the database world outside of MySQL, you know that Oracle is having a conference this week. It’s called Oracle Open World. Drips with irony doesn’t it? But this post isn’t about Oracle being open or otherwise.

This post is about the announcement being made Wednesday. It seems Oracle has a surprise. A pretty well kept surprise. It’s such a big deal that Larry Ellison himself is making the announcement.

Some people, including some of my colleagues at Pythian, are speculating that this is going to be an announcement about a share-nothing clustering solution.

In the first quarter of 2007, I interviewed with a company in Atlanta, seeking my first full-time job as a MySQL database administrator. They were an online company building a social-networking website with a virtual world interface (kind of like Second Life, from what I understood). They were using an (at the time) fairly unstable version of MySQL 5.1 only because it offered clustering with the ability to store data on disk while keeping the indexes in memory. Previously, in version 5.0, everything had to be stored in-memory. Much has improved with MySQL clustering since that time.

While I don’t know for certain that Larry is going to announce in-memory clustering, I kind of hope that is what it’s all about, because it would demonstrate this: Oracle is walking a trail blazed by MySQL.

Welcome to the 115^th edition of Log Buffer, the weekly review of database blogs.

I must thank Paul for taking over at last minute for LB#114 last week, when, as he put it, “ . . . a killer combo of painkillers and the pain that the painkillers can’t kill . . . ” reduced to me a less-than Log-Buffer-capable state. Or to be more precise, to a writhing, benighted gargoyle of misery. (Too colorful?)

Anyway, the good news is that I’m better. Not all better, mind you. Between the tooth thing and my spending all my working time on a special project, there was nothing left for poor old Log Buffer. So, I face the choice: throw it open to you, LB’s loyal readers for your contributions; or adopt Paul’s approach¹ from last week, and use the nifty AideRSS.

I’m going to bet on our readers. Let’s hear from you with your picks for best database blogs of the week gone by. I promise you a real, proper Log Buffer next week, from someone. If not me, well, Nick Westerlund still wants his go, and Ward Pond is back looking for a slot.

Until then, wish me luck with my angry tooth.

1. The truth is that I was briefly worried about having my job taken away by software. My concerns were allayed, at least partially, when I saw that the original software-built list of database blogs also included an item from “manscaping.com”, which I’m fairly sure had little or nothing to do with database administration.

This is the 114^th edition of Log Buffer, the weekly review of database blogs.

I am sorry to say that this log buffer was supposed to be edited by Dave Edwards, but he’s suffering from severe and long-lasting tooth pain and until his root canal is done he’s KO’d by a killer combo of painkillers and the pain that the painkillers can’t kill. I’ve been there myself, twice, and here’s a tip Dave. It hurts until the dentist takes out the needle. Then the pain goes away while he digs. The pain comes back that night. The next morning it’s worse than ever, unbelievably, writhingly bad. But later that afternoon, blisssssssssssss. :-) Good luck man.

This Log Buffer has been generated in a completely automated way with the help of the incredibly awesome AideRSS.

To give you an idea of just how awesome it is, I was able to load up Dave’s complete OPML file of all the blogs he monitors for Log Buffer. And AideRSS applied it’s magical PostRank algorithm which scores blog posts based on how many comments, del.icio.us bookmarks, blog links from other blogs, etc. that it received, along with some more secret sauce they don’t publicly tell us about (kind of like Google with their Pagerank equivalent). The number to the left of each headline represents the linked item’s AideRSS PostRank.

It did a great job of automatically selecting the best posts from the last week.

To give you an idea of AideRSS’s helpfulness, here are a couple useful feeds I suggest you subscribe to:

1. PlanetMySQL, but only with posts that rank “Best”
2. OraNA.info, again only the posts that rank “Best”. Note that there is a bug in Eddie’s feed that makes it impossible to use all possible information on the ranking.
3. SQLBlogs.com processed by AideRSS to show only the best posts.

While I have no idea how AideRSS plans to make moolah, I think we can agree that is some kind of awesome if you’re like me and can’t afford to miss a big story, but can’t afford the time you would need to read it all. Many thanks to Andrew Baldwin and although that’s the AideRSS about page there there’s a good pic of Andrew on that page. I first met Andrew at MySQLConf 2008 this spring and he’s a great guy and a great advocate for this service.

With no further ado or free advertising for AideRSS, here’s this week’s fully automated Log Buffer. We do not plan a fully automated Log Buffer to become routine but depending on the feedback we might adopt this approach whenever we have a last minute cancellation due to illness or what have you. So your feedback would definitely be appreciated, thanks.

10.0 - Random selection, with a bias ..

(more…)

Last week Giuseppe Maxia announced the Call for Papers for the 2009 MySQL Users Conference and Expo, and also announced that there would be an unconference, MySQL Camp, organized by me.

It’s true! Currently MySQL Camp is set to happen, though I am still working out details with Colin Charles and Giuseppe Maxia. We had originally talked about having MySQL Camp on Tuesday and Wednesday, but I would like to add Monday so that folks attending the conference who are not attending a tutorial have a choice on Monday. I am also looking into lunch options, since the conference venue does not have many options within walking distance.

There will be plenty of collaborative effort, of course, which will appear later on. If you have input, you can always e-mail me, or leave a comment here. Whether there is a topic you really want to see, or some logistical detail you always see that’s missed, I would like to hear about it.

(yes, I said I wanted you to give me your opinions!)