Hacking Open Source

Mar 9, 2008 / By Sheeri Cabral

Tags: , ,

Open Source means that the source code is open. There are many inferences that can be made from this, and many stereotypes that can be applied, but in the end, all it means is that you can read the source code as well as use the binaries.

I am not a developer. I fully admit this. What I mean by this is that my strengths do not lie in engineering new code to do things. Give me some Lego™s or some Wedgits™ and I look at the instructions to see what to build.

However, I am an engineer.

One of my team’s current tasks is to restore a backup (using InnoDB Hot Backup, and compressed) from a client’s production machine to a development instance…..

Weekly — thus we want to automate it……

And did I mention the development machine has 5 different MySQL instances on it, running all on 127.0.0.1 on different ports?

The steps are:
0) stop the desired mysql instance
1) move the development datadir and logs to a backup directory
2) copy the most recent production backup to the development machine
3) uncompress the production backup and apply any logs (with ibbackup)
4) start the desired mysql instance
5) check and celebrate

Being smart, and knowing my own limitations, I delegated the task to another team member. Remember, I am not a good developer! But my team member is smart, and he got everything going well, except step 3. The problem was with running:

ibbackup --apply-log --uncompress ./restore-my.cnf

See, somehow we needed to generate the restore-my.cnf with the right parameters. We couldn’t just use what the backup used, because there were 5 instances of mysql, so what would we use for things like the “datadir”? I could just make a soft link, changing what the directory should really point to given a config file for that instance. Options such as memory parameters, which ibbackup does not care about, must be the value of the development instance.

Today the client wanted us to restore one of the development instances, so I took on the challenge of figuring out how to generate the right restore-my.cnf. I figured that if we could get all of the option values of the mysql instance we’d be all set. If this were a one-instance mysql server, I would simply use mysqld --print-defaults and I would be all set. However, mysqld_multi does not have such an argument, and none of the following worked:

> mysqld_multi --print-defaults start 1
Unknown option: print-defaults
Error with an option, see mysqld_multi --help for more info.

> mysqld_multi --mysqld='/usr/sbin/mysqld --print-defaults' start 1

FATAL ERROR: Tried to start mysqld under group [mysqld1], but no mysqld binary was found.
Please add "mysqld=..." in group [mysqld_multi], or add it to group [mysqld1] separately.

> mysqld_multi --mysqld=/usr/sbin/mysqld --print-defaults start 1
Unknown option: print-defaults
Error with an option, see mysqld_multi --help for more info.

> mysqld_multi --mysqld=/usr/sbin/mysqld start 1 --print-defaults
Unknown option: print-defaults
Error with an option, see mysqld_multi --help for more info.

>

I will note for the record that mysqld=/usr/sbin/mysqld_safe was already specified in the my.cnf under the [mysqld_multi] section. And I tried all four of the above commands using /usr/sbin/mysqld_safe as well, no dice. I thought the commandline parameters were supposed to override those in files, but in this case that was not true empirically.

At any rate, what I needed was a list of every option in the options file, but only for the default options plus that specific instance. At the time, I was chatting with Mark Atwood (a friend whom I have known socially for years; I only found out he was writing storage engines when he showed up at the first MySQL Camp!). I started to ask him if he would make a perl script to get all the options — it would be pretty simple, just write a script that parses the /etc/mysql/my.cnf file and get every line except for the ones in the [mysql#] directives where # != my instance #.

Then I realized — hey, mysqld_multi has to do this already, because you can specify mysqld_multi start #. So I took the “easy” way out, and ran

cp mysqld_multi get_options.pl

and proceeded to muck with get_options.pl until it did what I wanted, namely print out

[mysqld]
option1
option2

optionLast

And dear reader, the whole point of this message was to promote the fact that had mysqld_multi not been an open source script, I would probably still be hacking at the perl.

Actually, probably not, but the point is that I was able to reuse the same code that MySQL uses, without having to come up with a hack. Writing a script to get the options when MySQL already has that code is a waste; reusing MySQL’s code ensures that I get the exact same output (all features/bugs included!) that the mysql instance itself would get.

And that is smart technology. With closed source programs I often find myself frustrated, saying “I know they are doing X because they get Y and now I want to get Y…..but I do not know how they do X because the source is closed!”

4 Responses to “Hacking Open Source”

  • Sergei Golubchik says:

    I’d probably use my_print_defaults for this. Something like

    my_print_defaults mysqld mysqld1 | sed -e ‘1i [mysqld]’ -e ‘s/^–//’

  • Sheeri Cabral says:

    You know, I had thought to try that, but I’d remembered wrongly that it was “mysql_print_defaults” and could not find the program.

    It would actually be

    my_print_defaults mysqld_multi mysqld1 | sed -e ‘1i [mysqld]’ -e ‘s/^–//’

    I do believe that if this were a closed source program, though, I would not even have access to the “my_print_defaults” program, which is basically part of the code library.

    The point remains: Being able to use the tools that the MySQL developers use is extremely valuable to the community, and it drives home that even though I do not hack the C code and probably never will, I am still benefiting from open source.

  • Arjen Lentz says:

    Hi Sheeri… mysqld_multi is broken in various ways and places, basically it has not been kept up-to date with other changes over the years.
    I’ve filed bugs on it. I tripped over this (again) a few weeks ago, and when checking in the bugs system I found that it was pretty much the same issue I’d filed as a bug about a year back. So I amended it. It’s in progress now, it seems.

  • […] Hacking Open Source […]

Leave a Reply

  • (will not be published)

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>