Pythian Group

The MySQL errorlog is an important point of reference when administering a MySQL Server. We can grasp much about the state of our MySQL instance by the informational and error messages written out to it by our MySQL daemon. Our monitoring suite is set up to check the mysqld  error log file periodically for any new nasties logged and then it alerts  us if there’s anything to know about. Recently I was asked to investigate some replication outage alerts a colleague had received overnight. One  of the primary directions I took was the error log file. This is where I would expect find any evidence of replication being stopped or crashes etc – I was looking for anything that could fill me  in on the causes of replication alerts. When I ran the command to tail the log I was shocked to see the log was totally empty. Read the rest of this entry . . .

As an editor (of http://www.mysqlzine.net), I cringe at the title of this post. However, it is absolutely accurate.  Recently, we had a situation where we had two servers running Sun Solaris 10 on some high-end Sun hardware.  I don’t remember exactly, but it was one of Sun’s upper-end boxes with AMD procs. Nice boxes, really. The two servers are configured in a master-master circular replication setup.

Here is the problem.  On both servers, the error logs were being created incorrectly.  On one of them, it was creating an error log that was 154 megabytes in size.  FLUSH LOGS worked, but the newly-created error log would be the same size. While there was some data in the file that I could use the cat, head or string command to discern, the majority of the file was not text data.

After working on this for a bit, I logged into the secondary server and discovered that the error log on this server didn’t look right either  — the same characteristics of large size with almost no actual text data.  The only difference is that these error logs were around 20 megabytes in size. I googled around a bit and couldn’t discover anyone with a similar problem.

I can’t figure out what was causing this.  We checked everything we could think of, and during some other maintenance, restarted the mysqld daemon. That didn’t work either — when the server came back online it was experiencing exactly the same problem with the error log. Finally, during hardware maintenance to upgrade the memory, the servers were rebooted.  The next morning, I checked them, and found both error logs working exactly as they should. However, it took that server reboot to fix the problem.

I am at a loss to explain what was wrong.  If anyone has any thoughts or a similar experience, I would love to hear from you!

Or, When MySQL Lies!

When I do a show slave status\G, sometimes mysqld will lie to me and give me a wrong Exec_Master_Log_Pos. Let me explain with a situation from last night.

This is the output of show slave status\G from mysql version 5.0.41-community-log:

mysql> show slave status \G
*************************** 1. row ***************************
                Slave_IO_State: Waiting for master to send event
                   Master_Host: XXX.XXX.XXX.XXX
                   Master_User: replic_username
                   Master_Port: 3306
                 Connect_Retry: 60
               Master_Log_File: mysql-bin.000480
           Read_Master_Log_Pos: 690470773
                Relay_Log_File: db2-relay-bin.000028
                 Relay_Log_Pos: 683977007
      Relay_Master_Log_File: mysql-bin.000480
              Slave_IO_Running: Yes
             Slave_SQL_Running: No
               Replicate_Do_DB:
           Replicate_Ignore_DB:
            Replicate_Do_Table:
        Replicate_Ignore_Table:
       Replicate_Wild_Do_Table:
   Replicate_Wild_Ignore_Table:
                    Last_Errno: 0
                    Last_Error: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MySQL code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave.
                  Skip_Counter: 0
           Exec_Master_Log_Pos: 126
               Relay_Log_Space: 690471192
               Until_Condition: None
                Until_Log_File:
                 Until_Log_Pos: 0
            Master_SSL_Allowed: No
            Master_SSL_CA_File:
            Master_SSL_CA_Path:
               Master_SSL_Cert:
             Master_SSL_Cipher:
                Master_SSL_Key:
         Seconds_Behind_Master: NULL

So in summary, the slave SQL thread is stuck (in this case because of a problem during the transfer of the binlog data to the slave’s relay log). The show slave status\G command tells me that it is stuck at the master binlog file mysql-bin.000480, position 126.

But, if I look at the error log file entries when the slave got stuck I see:

Read the rest of this entry . . .