Hacking Open Source
Mar 9, 2008 / By Sheeri Cabral
Open Source means that the source code is open. There are many inferences that can be made from this, and many stereotypes that can be applied, but in the end, all it means is that you can read the source code as well as use the binaries.
I am not a developer. I fully admit this. What I mean by this is that my strengths do not lie in engineering new code to do things. Give me some Lego™s or some Wedgits™ and I look at the instructions to see what to build.
However, I am an engineer.
One of my team’s current tasks is to restore a backup (using InnoDB Hot Backup, and compressed) from a client’s production machine to a development instance…..
Weekly — thus we want to automate it……
And did I mention the development machine has 5 different MySQL instances on it, running all on 127.0.0.1 on different ports?
The steps are:
0) stop the desired mysql instance
1) move the development datadir and logs to a backup directory
2) copy the most recent production backup to the development machine
3) uncompress the production backup and apply any logs (with ibbackup)
4) start the desired mysql instance
5) check and celebrate
Being smart, and knowing my own limitations, I delegated the task to another team member. Remember, I am not a good developer! But my team member is smart, and he got everything going well, except step 3. The problem was with running:
ibbackup --apply-log --uncompress ./restore-my.cnf
See, somehow we needed to generate the restore-my.cnf with the right parameters. We couldn’t just use what the backup used, because there were 5 instances of mysql, so what would we use for things like the “datadir”? I could just make a soft link, changing what the directory should really point to given a config file for that instance. Options such as memory parameters, which
ibbackup does not care about, must be the value of the development instance.
Today the client wanted us to restore one of the development instances, so I took on the challenge of figuring out how to generate the right restore-my.cnf. I figured that if we could get all of the option values of the mysql instance we’d be all set. If this were a one-instance mysql server, I would simply use
mysqld --print-defaults and I would be all set. However, mysqld_multi does not have such an argument, and none of the following worked:
> mysqld_multi --print-defaults start 1 Unknown option: print-defaults Error with an option, see mysqld_multi --help for more info. > mysqld_multi --mysqld='/usr/sbin/mysqld --print-defaults' start 1 FATAL ERROR: Tried to start mysqld under group [mysqld1], but no mysqld binary was found. Please add "mysqld=..." in group [mysqld_multi], or add it to group [mysqld1] separately. > mysqld_multi --mysqld=/usr/sbin/mysqld --print-defaults start 1 Unknown option: print-defaults Error with an option, see mysqld_multi --help for more info. > mysqld_multi --mysqld=/usr/sbin/mysqld start 1 --print-defaults Unknown option: print-defaults Error with an option, see mysqld_multi --help for more info. >
I will note for the record that mysqld=/usr/sbin/mysqld_safe was already specified in the my.cnf under the [mysqld_multi] section. And I tried all four of the above commands using /usr/sbin/mysqld_safe as well, no dice. I thought the commandline parameters were supposed to override those in files, but in this case that was not true empirically.
At any rate, what I needed was a list of every option in the options file, but only for the default options plus that specific instance. At the time, I was chatting with Mark Atwood (a friend whom I have known socially for years; I only found out he was writing storage engines when he showed up at the first MySQL Camp!). I started to ask him if he would make a perl script to get all the options — it would be pretty simple, just write a script that parses the /etc/mysql/my.cnf file and get every line except for the ones in the [mysql#] directives where # != my instance #.
Then I realized — hey, mysqld_multi has to do this already, because you can specify mysqld_multi start #. So I took the “easy” way out, and ran
cp mysqld_multi get_options.pl
and proceeded to muck with get_options.pl until it did what I wanted, namely print out
And dear reader, the whole point of this message was to promote the fact that had mysqld_multi not been an open source script, I would probably still be hacking at the perl.
Actually, probably not, but the point is that I was able to reuse the same code that MySQL uses, without having to come up with a hack. Writing a script to get the options when MySQL already has that code is a waste; reusing MySQL’s code ensures that I get the exact same output (all features/bugs included!) that the mysql instance itself would get.
And that is smart technology. With closed source programs I often find myself frustrated, saying “I know they are doing X because they get Y and now I want to get Y…..but I do not know how they do X because the source is closed!”
4 comments on “Hacking Open Source”
Pingback: Automating To Save Time