The MySQL Team

Being able to work side by side with two MySQL experts like Sheeri Cabral and Augusto Bott has been a great experience. I have learned a lot, not only about MySQL, but also about what a great DBA should be like. Both of them are recognized MySQL Community members and regular speakers at the MySQL Users Conference and other events. Sheeri has been named MySQL Community Member of the Year twice in a row. Both are a guarantee of excellent service.

My Predictions For Pythian

I believe that the IT services industry will undergo a change that is continuation of the the transformation that the IT industry as a whole has been going through over the last few years: commoditization.

Very few will argue at this point that hardware and software are commodities, especially when they support a company’s infrastructure. This trend has put MySQL in the position it has today in the database arena. In this scenario, managing the IT infrastructure, whether it’s running in The Cloud, a hosting company or internally, no longer adds value to the core business. As in any economic downturn, companies reconsider their overhead and look for solutions that are more flexible and cost-effective that their in-house services. This is when they start looking into outsourcing alternatives.

This time, however, is different—they have a designated fixed budget and very specific needs. The traditional consulting model doesn’t fit this scenario since it doesn’t have the required flexibility. This is where companies like Pythian come in.

Pythian’s business model and practices allows the clients to control their budgets while keeping a healthy infrastructure, and concentrating on the tasks that add real value to their business. Under the direction of Andrew Waitman, CEO, and Paul Vallée, Pythian’s founder, the company is in a an excellent position to satisfy the new market needs.

What’s Next

I will continue participating as actively as I can in the MySQL Community and continue with the development of sar-mysql. My blogs will move to a new home, but will be a continuation of the of the articles I have been writing here (without Dave Edwards cleaning up my English though); and they will also be syndicated in Planet MySQL. I will do my best to keep an eye on the articles I posted in the Pythian blog to keep answering any new comments that may be posted there. If you have the patience to deal with my non-technical tweets, feel free to follow me through Twitter: @seattlegaucho.

I will see you around!

P.S. to the MySQL DBAs in the wild: This might be a good opportunity to polish your résumé and submit it to Pythian. Who knows, you may end up working for a company on the rise, side-by-side with some of the best MySQL DBAs in the market.

The title of the Launchpad page already reflects the change. What remains to be done is: a) change the name of the Perl script and documentation; and b) change the Launchpad URL. It is likely that I will change the name of the script when I release version 1.x (see below). I’m not sure of all the implications in Bazaar regarding the URL change, so that task will have to wait for now.

Now a little more info on the status of the project.

Bug Fixes

In the latest trunk there are two patches. One corresponds to Bug #455870, which should be fixed. I’m excited since this bug was posted by a user. (Yay!)

The 2nd patch refers to a bug that came up during an implementation in a client, in a master-slave configuration (in hindsight, I should have thought about it earlier). The snapshots from the master were being replicated to the slave, so when the script ran on the slave, the autoincrement values were in conflict and stopped replication. In the latest patch (build 16), I have added a column that records the server id, as in SHOW GLOBAL VARIABLES LIKE 'server_id' to distinguish the master and slave data. I like the fact that it is possible to query the master and slave snapshots side by side to diagnose slave lag. This change makes the last version in the trunk incompatible with the newer ones, which triggered the creation of the first README file.

I will be posting a new tarball soon.

Road Map

Believe or not, there are enough ideas in the queue to justify a road map. Here’s a summary list and you’re welcome to comment on it. I promise to review all the suggestions.

Code Review

As I mentioned above, the script currently works, but it’s far from optimal. So I’ll be changing the code base with two objectives. 1) Increase overall efficiency; and 2) facilitate future improvements. The more profound code changes will be implemented in version 0.x (current version). I believe that messy code leads to bugs, so the cleaner it is, the less likely I’ll break existing functionality while adding new one.

Command Line Syntax Changes

Adding new functionality implies that the script will perform more tasks which in turn will add overhead to each one of them. To minimize this overhead, I am working on a different command line syntax and the corresponding underlying code. This change will happen in version 1.x (the next version) since the code has to be cleaned up to be effective and works as it is. The command line syntax change will imply rewriting any wrapper script that might be in place to invoke in the crontab, so this will be the opportunity to rename the script as well.

Installer

The script is so simple that distributing a tarball is enough for now, but I’d like to have an installer and possibly a package that will take care of the Perl modules dependencies, schema initialization and user credentials verification / creation. I have no target version for this. I’ll shoot for 1.x, but it’s likely to go into 2.x.

Other

I’m still working on plenty of use cases, best practices, and companion scripts examples. In time, the example scripts and documentation will be part of the complete package and installed along side with the main utility. In the meantime, just follow the blogs and the links to them in the Launchpad announcements page.

Community Participation

Having a user file a bug, Schlomi’s bringing up the name issue, and Sheeri contributing with her sanity checks are a great help, but I invite all of you out there to participate through comments to my blogs, bug reports, questions in the Launchpad page, and replying to my tweets.

I will be participating in OpenSQL Camp in Portland, OR next month. I’m sure I will have an opportunity to review my ideas with the old friends I’ll be meeting there. Stay tuned.

Oracle – Oracle Open World 2009

There is so much material about OOW09, that I’m giving a full subtitle to it.

Let’s start with a quick recap of the keynotes by Scott McNealy and Larry Elison in this article by Andrew Clarke: The return of The Scott And Larry Show. The recap suggests that the presentations aimed to show how Sun & Oracle (aka Team Red) would challenge IBM (aka Big Blue) head-on. Larry didn’t fail to mention Oracle’s intention to invest in MySQL.

Of course the conference wasn’t short on technical issues, and these articles prove it. Jason Arneil talks about 11 Things about 11gR2. Chen Shapira discusses one of the lessons she learned, and shares with us some Shell Script Tips.

Apparently one of the most exciting event was the Bloggers’ Meetup, and Pythian’s T-shirts—which had to be signed by the attendees to win a netbook—were a great success. You can read all about it in Doug Burns’s article OOW 2009 – Blogger’s Meetup (don’t miss the links at the bottom for more of his articles); and in Richard Foote’s Day 3 Highlights. And of course our own Alex Gorbachev’s blog Pythian OOW09 Diaries: Bloggers Meetup with plenty of pictures.

Last but not least is Oracle Magazine’s DBA of the Year: Husnu Sensoy. Congratulations from the folks at Pythian, Hunsu!

Oracle

Let’s start with Chris Foot’s The Art of Being a Successful DBA – Finding Information Quickly.

I agree 100% with all Chris’s criteria, so here are my links for the MySQL DBAs: 1) Planet MySQL or it’s Twitter counterpart @mysql_community, 2) MySQL online documentation (remember to read the manuals), 3) MySQL Forums & 4) Twitter’s #mysql RSS feed. I have more resources bookmarked, but these are a great starting point. Jonathan Lewis’s article Experts is somewhat related and has great advice as well.

How many DBAs face the question of recovering data that has been accidentally deleted? Luis Moreno Campos presents Reversing the effect of a TRUNCATE TABLE in Oracle 11gR2. And still on typical DBAs’ tasks, we also have articles by Richard Foote: How To Rebuild And Make An Index Bigger, Not Smaller (Carry That Weight), Miladin Modrakovic’s Blocking locks history, Marco Gralike’s HOWTO: Partition Binary XML, XMLType Storage and Slavik Markovich’s important recommendation on security Blind SQL Injection in Oracle. The latter one has Oracle in the title, but I believe the concepts he discusses can be applied to any database.

MySQL … Or Should That Be Oracle?

Monty Widenius, one of the three founders and creators of MySQL wrote an interesting press release concerning Oracle/Sun explaining why he believes that the European Unionconcerns about Oracle owning MySQL are justified. This created quite a stir in the MySQL community, with many people writing their own open letters, recommendations, and editorials on the matter.

Zak Greant’s Letter to the EC on the Oracle/Sun Takeover, Lukas Kahwe Smith’s MySQL needs to be saved from Oracle? and Brian Aker’s RMS, GPL, The Peculiar Institution of Dual Licensing are just a sample of what’s out there. If you are interested in the subject, I recommend you browse the articles in Planet MySQL.

In my opinion, I don’t believe that the community and customers should be concerned—there are enough companies in the ecosystem with MySQL branches and and/or providing services, that any Oracle action to de-activate or damage MySQL would be pointless. The only companies that could be affected, are those who use MySQL embedded in products which are not released under GPL (OEM customers). These companies hold commercial licenses, but could still open their software under GPL or compatible licenses and concentrate in their core values to avoid any issues that Oracle may cause.

MySQL

To finish on a high note, I’d like to highlight a few good MySQL-related articles that I came accross while preparing this blog.

Percona has published two good articles that help us to understand some of the decisions that a typical MySQL DBA has to face sooner or later. The first article describes when storing the data in the DB might not be a good idea and memcached would, in MySQL-Memcached or NOSQL Tokyo Tyrant – part 1. The other article discusses different High Availability strategies to use with MySQL in Finding your MySQL High-Availability solution – The questions with plenty of comments. Kris Buyaert decided to comment on the article in a blog of his own: Nines , Damn Nines and More Nines.

Finally, Lenz Grimmer invites us to submit talks for FOSDEM 2010 in Brussels, Belgium: FOSDEM Call For Participation opened – submit your talks now!. This might be a good excuse to visit Europe.

I hope the SQL Server folks don’t feel dissapointed that we left them out on this edition, but Oracle Open World and Monty’s press release gave us too much material to sift through. I’m sure Dave will come up with plenty of cool highlights for you on the next edition of the Log Buffer.

What Was Going On Around 2:30pm?

This is a question a customer asked us. To answer it we ran MySAR for a few days and queried the results for analysis. Looking at the data, we determined that the number of INSERT operations was significantly higher than any other, so we queried for the Com_insert status values. Com_insert is a counter that accumulates the number of INSERTs issued since the last server start (or since the last FLUSH STATUS command). For details on the variables available check Chapter 1. mysqld Options/Variables Reference.

The initial query we used was:

select value_stat_name name, value_stat_timestamp ti, value_stat_number val, value_stat_delta delta from value_stat
where value_stat_name like "Com_insert"
and HOUR(value_stat_timestamp) between 13 and 15
and date(value_stat_timestamp) between "2009-08-22" and "2009-08-25"
order by 1, 2
+------------+---------------------+-----------+-------+
| name       | ti                  | val       | delta |
+------------+---------------------+-----------+-------+
| Com_insert | 2009-08-22 13:00:01 | 163777289 | 16051 |
| Com_insert | 2009-08-22 13:10:01 | 163794262 | 16973 |
| Com_insert | 2009-08-22 13:20:01 | 163810327 | 16065 |
| Com_insert | 2009-08-22 13:30:01 | 163826242 | 15915 |
....
| Com_insert | 2009-08-25 15:20:01 | 168362221 | 13835 |
| Com_insert | 2009-08-25 15:30:01 | 168376888 | 14667 |
| Com_insert | 2009-08-25 15:40:01 | 168392410 | 15522 |
| Com_insert | 2009-08-25 15:50:02 | 168408476 | 16066 |
+------------+---------------------+-----------+-------+
72 rows in set (1.56 sec)

72 rows of numbers are not very easy to interpret, so the next step was to create a graphic to visualize the results. Taking the ideas from Shlomi Noach’s blog, SQL pie chart, I decided to try and create the charts using the Google Charts API.

Easy Graphics How-To

The first step was to create the string of values needed for the chart. I achieved this using GROUP_CONCAT, adding some formulas to scale the values properly:

select day(value_stat_timestamp) as day,
GROUP_CONCAT((value_stat_delta-12000)/70) as series from value_stat
where value_stat_name like "Com_insert" and HOUR(value_stat_timestamp) between 13 and 15
and date(value_stat_timestamp) between "2009-08-22" and "2009-08-25"
group by day(value_stat_timestamp);
+------+------------------------------------------------------------------------------------------
| day  | series
+------+-----------------------------------------------------------------------------------
|   22 | 57.8714,80.3000,81.2571,94.3857,85.2571,66.0857,94.6429,74.2857,91.1000,...
|   23 | 74.4857,95.5857,103.4571,101.3000,75.9429,75.3571,79.3000,83.0571,94.7714,...
|   24 | 25.6286,38.8429,31.7000,36.3857,50.0143,44.3286,46.1571,39.4714,34.3857,21.5571,...
|   25 | 6.8000,12.9714,33.0429,13.8286,26.2143,38.1000,50.3143,19.4429,13.0857,3.2143,7.9143,...
+------+------------------------------------------------------------------------------------------
4 rows in set (0.34 sec)

(To make the output more readable, I’m not including the whole lines.) Using the values corresponding day 22, the most simple line graphic can be created with following HTML tag:

<img src="http://chart.apis.google.com/chart?cht=lc&chs=400x200&
chd=t:57.8714,80.3000,81.2571,94.3857,85.2571,66.0857,94.6429">

Resulting in:

In the URL string, cht, chs, and chd represent chart type, chart size, and chart data respectively; you can check the details in the Google Chart API URL I quoted above.

You can copy and paste the <img> tag above into an HTML document (no need to add any text) and open it with any browser. For Firefox, the URL to open a file on disk would look similar to this:

file:///path/to/file/file_name.html

Go ahead, open any text editor, copy and paste the example above in any file, save it with extension .html and open it with your browser.

The chart with the complete data set and and some decorations looks like this—each color represents a different day from the dataset above:

Using the function CONCAT,  it is possible to create the complete <img src=”…”> tag directly from the SQL statement by adding the proper sub-strings.

Conclusion

Using MySAR, it is possible to to collect the data, query the data for diagnosis. and create graphics to visualize those values.

By the way, there was nothing significant happening in the database at 2:30pm.

A couple of months back, Tim Procter, Sheeri Cabral and I were discussing about how best to diagnose a MySQL server and/or tune its performance, automating the process as much as possible. The Performance Advisors from MySQL Enterprise do this, but most of our customers don’t have a subscription and Pythian’s collective experience is not necessary reflected by its rules.

In our daily work, we have used Major Heyden’s MySQL Tuner, Mark Leith’s Statpack and our own tools to review a MySQL server configuration parameters. However, all of these tools had limitations in regards of what we wanted to achieve. Our major concern was the need to keep historical data to draw conclusions based on trends and cross check with other time based tools. The conclusion of our conversations was that whatever scripts we would implement, it would need to monitor all configuration and status variables, and their evolution over time.

One day, while looking into slave lag problem, the idea of MySAR popped into my head and a few hours later I was using its first incarnation. I was able to relate the server’s I/O activity peaks with these lags and in turn, discovered that it was caused by a great number of INSERT statements coming in in waves. It was an encouraging outcome for what was nothing more than a proof of concept.

MySAR’s Goal

Major Heyden’s MySQL Tuner, Mark Leith’s Statpack, and our own scripts already do analysis and recommendations. Not wishing to reinvent the wheel, I decided that the best thing to do was to gather as much data as possible and store it in the database, from which it can be extracted to perform any kind of manipulations. My intention, then, is, to limit MySAR to be just a sampling tool. In short, to keep it simple.

What Does MySAR Do?

MySAR does nothing else than record the output of SHOW GLOBAL STATUS, SHOW GLOBAL VARIABLES, and SHOW FULL PROCESSLIST commands — or a configurable subset of their output. The idea is to invoke MySAR from the crontab (or similar tool on Windows) at regular intervals. Data older than 30 days is purged by default to avoid accumulating too much data on disk. We usually run it every ten minutes so it will be roughly in sync with sar’s output.

Where to Get MySAR and Submit Feedback

I have decided to use Bazaar and Launchpad to track the project: http://launchpad.net/mysar. I found it to be the most convenient alternative. Consider the code to be in the alpha stage—not necessarily feature-complete and with bugs. The trunk branch has the most stable version (the current development branch has been merged into the trunk).  But if you just want the latest public tarball, you can use the download link in the project Launchpad page.

Documentation

The documentation is provided in plain text, HTML, and perldoc formats (perldoc mysar.pl) as part of the tarball. It is complete since I use it as a specification document and gets written before the code is complete. You can download the HTML manual (click the “download this file” link on that page).

Please record bugs, questions, suggestions in the appropriate sections of the launchpad site and I’ll address them as best I can. I invite the community at large to contribute with use cases and examples in the Answers section, and I’ll post the most creative contributions as FAQs (along with the real FAQs). I don’t expect too much in the beginning. Hopefully the contributions will start coming in once it proves as useful as it has been for us so far.

Installation

At this point the requirement are only the DBI and DBI::mysql Perl modules. To install follow these steps:

1. Download the latest tarball from the download link on the Launchpad page.
2. Untar on any subdirectory using:

   tar -xzf mysar.tar.gz

3. Create a schema to hold it’s tables (or use an existing schema with the –database option). The default schema name is mysar. Example:

    CREATE DATABASE mysar;

4. Make sure the user already exists and has full access to this schema. Example to grant the permissions:

   GRANT ALL ON mysar.* TO 'username'@'localhost'

5. Run at least once to create the tables. The mimimum command line required is:

   mysar --user username --pass password --status / variables

   Either –status or –variables should be used. Currently there is a bug (Launchpad’s bug #441006) when you run with –status for the first time, which can be safely ignored.

I will be blogging different use cases based on real cases in the near future, including how to easily draw colorful graphics using the Google Charts API, so stay tuned.

MySQL

Johan Andersson explains in a very simple way the scenarios in which you may fall into a split brain situation and how to avoid it in MySQL Cluster on two hosts – options and implications. An article worth reading from one of the MySQL Cluster experts.

I love simple scripts that solve complex problems. I love it even more when the command line can be defined in an alias, and SQL from SQL offers some powerful examples. Ronald Bradford contributes another example in MySQL DML stats per table. It would be fun if someone would collect these and post them somewhere as a script to add at the end of our .profile or .bashrc files.

Sarah Sproehnle wrote two interesting articles this week. The first one explaining in simple terms when and how to use MySQL’s Query Cache in MySQL’s query cache and the other one about how to overcome one of MySQL’s partitioning limitations in Partitioning by dates: the quick how-to. 

The MySQL@Facebook blog always offers very deep technical articles and these 2 on InnoDB I/O are no exception: When does InnoDB do disk IO? and MySQL@Facebook: Too much IO?. I wonder when I will have some time to test the Facebook patch for MySQL.

SQL Server

Database Programming: Did You Know IN Can Do This? … the title says it all, you’ll have to read this short article to find out.

Ben Nevarez offers an insight into SQL Server’s optimizer in How the Query Optimizer Uses Statistics. Another article worth reading is Spools in Execution Plans. Understanding how the optimizer works in your databases server is the key to write good SQL statements.

I have two pet peeves when it comes to the IT Industry practices: 1. Lack of attention to details; and 2. wasted resources. In Can you be too Thorough? Jason Massie talks about being thorough (a way to have attention to detail), and knowing when to stop in a project (a way to avoid waste of resources). Thomas LaRock offers another twist on the subject with Reproduce, then Repeat.

Oracle

Oracle 11gR2 has been released, so it’s time to start looking into the details. Christian Antognini’s Deferred Segment Creation and his Script to Download 11gR2 Documentation are useful articles to get going. If you need help upgrading, read Get Upgrading: Steps To Upgrade To 11gR2 by Saurabh Sood.

Analytical functions can be a pain in all databases. The detailed article from Jonathan Lewis — Analytic Agony looks into ways to alleviate that pain.

Since Oracle 10.2 it’s valid to say that buffer cache can be stored inside shared pool. If you are curious about this fact, Tanel Poder delivers the proof in KGH: NO ACCESS allocations in V$SGASTAT – buffer cache within shared pool!. Heed the warning and do not run the examples in a production environment unless you want a posse of users hunting you down your company hallways.

Did I mention already how much I like simple shell tricks? Jared Still offers some Shell Tricks for the Oracle fans.

Last but not least, another article addressing attention to detail on which Chris Foot gives it a funny name: Poka-Yoke. You’ll have to read The Art of Being a Successful DBA – Poka-Yoke and Paranoid DBA Best Practices to know what it means.

Suppose you have a MyISAM table containing a column with a full text index. This table starts to grow to a significant size (millions of rows) and gets updated fairly frequently. Chances are that you’ll start to see some bottlenecks when accessing this table, since without row level locking, the reading and writing operations will be blocking each other.

A solution that many people would suggest right away is to use the master for writes and a slave for reads, but this only masks the problem, and it won’t take long before enough read traffic on the slave starts causing slave lags.

Why Sphinx?

The main difference between the Sphinx search engine and other alternatives is its close integration with MySQL. For example, it can be used as a storage engine.  In this way, Sphinx’s impact on existing application code can be minimized, opening the door for its more advanced features in future releases.

Initial Considerations

Need to compile MySQL from source

I struggled with this one for a while, until I realized that the storage engine plugins only work if the full version number of the MySQL server matches the full version number of the MySQL libraries used in the compilation; and if compiler options for the server, libraries, and plugin also match.

I couldn’t find any other reliable way of getting everything to work together than grabbing a MySQL source tarball and compiling it with Sphinx SE. The process is straightforward if you follow the procedures in the MySQL’s manual: 2.10.1. Source Installation Overview.

Use MySQL 5.1.xx

Although it is possible to use Sphinx with 5.0.xx, I like the cleaner plug-in storage interface of 5.1.xx.

The table to be indexed needs a unique INT column

The table for which you are going to create a full text index has to have an INT column declared as UNIQUE KEY. This value is the one to which Sphinx has to point the rows containing the text in its indexes.

Putting everything together

Both Sphinx and MySQL have very clear and straightforward instructions on how to compile from the source code tarballs, so just follow the steps. Although I had to run several options until I got all the right pieces in place, compiling either tool was never an issue.

The instructions for installing Sphinx can be found in its online documentation 2.3. Installing Sphinx on Linux; and to compile into MySQL 5.1 in 7.2.2. Compiling MySQL 5.1.x with SphinxSE.

When Sphinx is installed, it has two different sample configuration files. One is very comprehensive and the other is clean, simple, functional, as billed in its header. Use the latter one to get started.

Example run

As always, I use the Amarok player MySQL database as an example and in this case, I used the lyrics table.

In the sphinx.conf file I used the following line to index the table:

	sql_query			= \
		SELECT lid, url, lyrics FROM lyrics

Notice that with this configuration line, I’ll be indexing both the song title (the file name, really) and the lyrics text.

After I added an INT AUTO_INCREMENT PRIMARY KEY column to the lyrics table and converted it into InnoDB, the resulting table was:

show create table lyrics\G
*************************** 1. row ***************************
       Table: lyrics
Create Table: CREATE TABLE `lyrics` (
  `lid` int(11) NOT NULL AUTO_INCREMENT,
  `url` varbinary(255) DEFAULT NULL,
  `deviceid` int(11) DEFAULT NULL,
  `lyrics` text,
  `uniqueid` varbinary(32) DEFAULT NULL,
  PRIMARY KEY (`lid`),
  UNIQUE KEY `lyrics_url` (`url`,`deviceid`),
  KEY `lyrics_uniqueid` (`uniqueid`)
) ENGINE=InnoDB AUTO_INCREMENT=110 DEFAULT CHARSET=latin1

Once the table was in place, I ran Sphinx’s indexer utility to create the full text indexes.

Then I added the SPHINX table as per the documentation:

show create table sphinx_search\G
*************************** 1. row ***************************
       Table: sphinx_search
Create Table: CREATE TABLE `sphinx_search` (
  `id` int(11) NOT NULL,
  `weight` int(11) NOT NULL,
  `query` varchar(3072) NOT NULL,
  KEY `query` (`query`)
) ENGINE=SPHINX DEFAULT CHARSET=latin1 CONNECTION='sphinx://localhost:3312/test1'

A difference from the examples in Sphinx’s documentation is that I left out any additional attributes. And finally, I created a view to hide as many implementation details as possible

CREATE VIEW `vlyrics` AS select `ly`.`lid` AS `vlid`,
            `ly`.`url` AS `vurl`,
            `ly`.`deviceid` AS `vdevicedid`,
            `ly`.`lyrics` AS `vlyrics`,
            `ly`.`uniqueid` AS `vuniqueid`,
            `ss`.`query` AS `vquery`
FROM (`sphinx_search` `ss` join `lyrics` `ly`) WHERE (`ss`.`id` = `ly`.`lid`);

Last but not least, I found out how many songs have the word “death” in their lyrics (I was going to use “love”, but it sounded too lame):

select vlid, vurl from vlyrics where vquery = 'death';
+------+------------------------------------------------------------------------------------------------------------+
| vlid | vurl                                                                                                       |
+------+------------------------------------------------------------------------------------------------------------+
|   60 | ./opt/musica/iTunes Music/Genesis/The Lamb Lies Down on Broadway Disc 2/03 Anyway.wma                      |
|   66 | ./opt/musica/iTunes Music/Compilations/Evita_ Premiere American Recording (Disc/1-04 Buenos Aires.mp3      |
|   88 | ./opt/musica/iTunes Music/Lake_&_Palmer_Emerson/Works__Vol._1/06_-_Closer_To_Believing_(Album_Version).mp3 |
+------+------------------------------------------------------------------------------------------------------------+
3 rows in set (0.04 sec)

Notice that in this case the the character “=” doesn’t mean “equals”, but “contains”. If I had renamed the original table and used its name for the view, this would have been the only change as opposed to using “like”:

select lid, url from lyrics where lyrics like '%death%';
+-----+------------------------------------------------------------------------------------------------------------+
| lid | url                                                                                                        |
+-----+------------------------------------------------------------------------------------------------------------+
|  60 | ./opt/musica/iTunes Music/Genesis/The Lamb Lies Down on Broadway Disc 2/03 Anyway.wma                      |
|  66 | ./opt/musica/iTunes Music/Compilations/Evita_ Premiere American Recording (Disc/1-04 Buenos Aires.mp3      |
|  88 | ./opt/musica/iTunes Music/Lake_&_Palmer_Emerson/Works__Vol._1/06_-_Closer_To_Believing_(Album_Version).mp3 |
+-----+------------------------------------------------------------------------------------------------------------+
3 rows in set (0.02 sec)

Note: disregard the times for each query—the table has only about 100 rows.

Conclusion

Implementing SphinxSE into MySQL proved to be easier than it seemed in the beginning, although it took some time to compile and install everything. With a creative use of views, it could potentially be implemented right away in legacy applications, offering numerous advantages.

Sphinx offers many additional features that I do not consider here. I recommend looking into the documentation to implement the more sophisticated indexing mechanisms and search methods.

This evaluation was done with Sphinx v0.9.8, which is labeled as stable since it was done for a customer project. Version 0.9.9, the current release candidate, has an new feature: it supports the MySQL communications protocol. This makes it possible to query text search engine directly using MySQL’s client libraries, which opens a new set of interesting possibilities.

Description

I am trying to test SphinxSE as a plugin instead of getting it statically linked and came across an annoying bug. When using the configure --with-plugins option only once, the engine is statically linked. When using it twice, the first engine is created as a plugin, and the 2nd one is linked statically. Here are a couple of examples:

./configure --prefix=/usr/local/mysql-5.1.33 --with-plugins=innobase --with-plugins=sphinx

plugin_innobase_shared_target='ha_innodb.la'   <-- plugin
plugin_innobase_static_target=''
plugin_sphinx_shared_target=''
plugin_sphinx_static_target='libsphinx.a'       <-- static
./configure --prefix=/usr/local/mysql-5.1.33 --with-plugins=sphinx --with-plugins=innobase

~/src/mysql-5.1.33$ egrep "plugin_(innobase|sphinx)" config.log
plugin_innobase_shared_target=''
plugin_innobase_static_target='libinnobase.a'      <-- static
plugin_sphinx_shared_target='ha_sphinx.la'         <-- plugin
plugin_sphinx_static_target=''

The bug is filed as Bug #45440.

Although I wrote a comment on the latter one, there is one point I thought was worth its own blog.

GRANT … IDENTIFIED BY PASSWORD…

You can work around having to specify the password in the open following these steps:

1. Use a local or non-public instance (for example using MySQL Sandbox) to define the user / password combination you need:

   CREATE USER 'name'@'host' IDENTIFIED BY 'secret';

2. Use SHOW GRANTS to get the hashed password:

   SHOW GRANTS FOR 'name'@'host';
   +------------------------------------------------------------------------------+
   | Grants for name@host                                                         |
   +------------------------------------------------------------------------------+
   | GRANT USAGE ON *.* TO 'name'@'host' IDENTIFIED BY PASSWORD '*<hash value>'   |
   +------------------------------------------------------------------------------+

3. Log in to the production (or public) server and use CREATE USER with the hash value obtained above:

   CREATE USER 'name'@'host' IDENTIFIED BY '*<hash value>'

This way, even if the CREATE USER ... (or GRANT) shows in plain text in the MySQL history, binary logs or anywhere else, the password remains secret.

MySQL was in many ways a leader for the Open Source industry, both in the way the way it marketed OSS and in the way MySQL the company was built. It’s the latter aspect that worries me. MySQL was a world pioneer in having a global workforce and this shaped both its culture and products.

When I joined MySQL in 2001, 90% of the company was distributed outside their native Swedish offices. (When Sun bought it, it was down to 80% outside the Cupertino headquarters, which I’m sure is still more than any large or middle-size corporation can claim.) This meant that they hired the best talent they could find wherever they found it, and that they could follow the talent wherever it went. During my years at MySQL, many people moved around from country to country without affecting their jobs. This diversity and flexibility shaped everything the company did, and indeed, the different backgrounds of those involved greatly influenced the characteristics of MySQL AB’s products and services themselves.

Sun was already deep into distributing its workforce when they acquired MySQL, so of all the big corporations, it was probably the one that best fit MySQL’s culture. Oracle, however, is a very different story, and I’m not sure how the distributed workforce that the Sun and MySQL employees are used to, will fit into the new organization. This change will affect MySQL’s future—I’m not sure how, but it will be very different from the MySQL we know.

(On a side note, although I don’t think Oracle is buying Sun because of MySQL, it looks like the Twitter community thinks so: both #oracle and #mysql are on the top 10 topics right now. Check both Twitter feeds, if nothing else, they’re fun.)