I’m pleased to announce that there will be the formal launch of Pythian Europe at the premises of the Canadian Embassy in Prague on Wednesday the 6th of August from 17:00 to 18:30. This historic event will be announced by Mrs. Sameena Qureshi, Trade Counsellor, Embassy of Canada; and Paul Vallée, President and Founder, The Pythian Group. Present will be various members from the press (IT and Business), as well as representatives from Oracle and Sun Microsystems, the Canadian Chamber of Commerce in Prague, and many more. We will prepare some unusual and very tasty snacks and refreshments.

We would love for readers of this blog to join us, so please consider this your special, personal invitation from me. Please come if you’re in Prague on Wednesday. If you plan to attend, please contact Dan at elbl@pythian.com.

I am traveling to Europe next week to brief major prospects in Germany (Daimler, MAN) as well as to attend to administrative matters at Pythian Europe in Prague and would love to meet any readers of this blog during this trip!

I’m especially interested in meeting:

• DBAs, Applications Administrators and Systems Administrators,
• Potential customers (IT Directors, DBA Managers, Supply Managers for IT), and
• Potential partners (IT product of service companies that could partner with Pythian to delight our mutual customers)

Here is my itinerary:

• Sunday, August: Frankfurt,
• Monday, August 4: Stuttgart,
• Tuesday, August 5: Munich, and
• Wednesday, August 6 through Saturday, August 9: Prague, Czech Republic.

Please reach out to me using vallee@pythian.com if you would like to meet!

I found myself, as a fresh member of The Pythian Group, losing precious moments just to change a few standard administrative settings on my new laptop with Microsoft Windows Vista. Having found the answers, I’m sharing them with you so that you can save some time, or spend it more pleasurably out in the summer.

User Account Control (UAC)

Too many prompts and confirmations for admin tasks? Set “Elevate without prompting” for “User Account Control: Behavior of the elevation prompt for administrators in Admin Approval Mode” under Start -> Settings -> Control Panel -> Administrative Tools -> Local Security Policy -> Security Settings -> Local Policies -> Security Options. Ref: UAC on Wikipedia.

Windows Vista Features

By default, telnet is not installed with Windows Vista. To (re-)enable it, select the Telnet Client check box in Start -> Settings -> Control Panel -> Programs and Features -> “Turn Windows features on or off”. Then, Windows Features dialog -> Telnet Client. Now you may use telnet from the command prompt as you are used to in XP. Ref: FAQ on windowshelp.microsoft.com

(more…)

Usenix 2008 - Automated System Management, by Æleen Frisch of Exponential Consulting (and numerous books)

What is automation?

generic [perl|shell] scripts with cron,at

Problem: overlap of effort

So folks developed automation systems. General automation tools are around:

cfengine, puppet, cfg2

These are general — files, directories, etc. Don’t need to use chmod and chown and underlying commands.

However, they don’t really survive reboots well. For that, we tend to use tools more towards jumpstart, kickstart.

Monitoring with Nagios, related tools are rrd-tools such as cacti, cricket, munin, “or any of 8,000 others.” Automating ideas like iostat.

Nessus is a security testing tool.

homegrown, general, performance related, also automated backups — bakula, amanda, legato.

What do you want automated?

“Coffee machines”.

A lot of unsolved problems are human interaction.

Other problems solved — using remote power management.

Inventory management is another issue. HP OpenView is one, but Frisch says folks are not happy with it. You can pay for high-end monitoring systems.

A question came up about an inventory of users on systems. LDAP or NIS or Active Directory is the traditional solution where there are no local accounts. There’s authentication and then authorization, and the automated tools usually have authentication information but not authorization information. (You can handle it, but making groups on these tools is usually painful.) Authorization is usually handled either locally or as “if you’re authenticated you’re authorized”.

We talked about how to power down 500 machines when the air conditioning goes out, or when the power is going down. Combinations of temperature probes, “wake-on-lan”, remote power on and off were discussed.

What do people use to automate installs and configuration on Windows? For installation, the Windows native tools are great. It was noted that efs works better on Windows.

Anyone using Splunk with Windows? One answer — it works OK, there are some daemon tools to convert Windows Event Log to syslog.

Splunk came up as a topic of discussion, how it’s a great log management software and solves a problem we’ve had for decades — how to deal with logs. Frisch says, “Splunk is the most promising thing out there.”

Record keeping of time was brought up, as well as time management. Basically what we do at Pythian, so I explained how we do things. Other folks brought up ticketing systems as well. Jira and RT (Request Tracker) and OTRS (Open Ticket Request System) were brought up as well.

Also for change management, some folks use ClearCase (not open source), and others use rancid, others use Trac or bugzilla + change management system like subversion. Jira was recommended as a product that does both (with an add-on).

Use DHCP to help automate IP assigning. rsync is your friend too.

(it occurs to me that a dishwasher is an interesting problem; why do we have a dishwasher instead of just having a sink/dishwasher hybrid? Similarly, a hamper that does laundry for you when it’s full.)

Or, How to Be a Command-Line Commando

Does it surprise you to learn that I’m a Linux guy? I’ve been using Linux, to the exclusion more-or-less of everything else, since about 1999. In the past, I’ve done a little programming and some junior system administration. I’m even LPI-certified.

With this background, I’m quite comfortable working in the shell (AKA the command-line), the natural habitat of the sysadmin^[1]. I frequently open a shell to do some quick work, and when I do, I use GNU’s Bash, which is the default on most Linux distributions. (I believe it’s also the default shell in Mac OS X.)

One of Bash’s features is editable command-line history, which makes your current command-line and its entire history available to you as an editable buffer. That offers a great way to streamline your work in the shell.

I suspect, however, that many shell users don’t even know about this better way. And it baffles me that many SAs I have seen in action — including some of Pythian’s own — don’t use this. They almost seem to prefer unnecessary effort — smashing away at their keyboards, repeating themselves, deleting with the Backspace key, scrolling, forwarding their cursor one character at a time, copying and pasting with the mouse, and so on. That’s a lot of elbow grease.

With Bash, or any other shell that uses the GNU readline library, you can use the following Emacs-like key-chords to make your life better. The point of this (as with so many things sysadmins and programmers do) is to save you effort, viz. typing. These aren’t all of them; they’re the ones I use:

(more…)

Fedora 9 was released on the 13^th. I waited a whole three days to make sure I wasn’t going to be the beta-tester. Then I tried out the live release, and finally decided to upgrade my main workstation to Fedora 9 today. To be sure I wouldn’t mess stuff up, I used the DVD installer to upgrade.

The upgrade finished fine, but when I rebooted, XChat would not run.

[14:15:53]$ xchat
xchat: error while loading shared libraries: libperl.so: cannot open shared object file: No such file or directory

What did rpm have to say about this?

[14:26:31]$ rpm -q xchat
xchat-2.8.4-11.fc8.i386

[14:26:39]$

Hmm . . .  looks like the package did not get upgraded. Wondering how many others did not get upgraded, I did a quick check:

[14:26:39]$ rpm -qa | grep fc8|wc -l
139

Wow! 139 packages still carry the fc8 tag! Next I checked if the updates repository has the updates:

[root@rajlin ~]# yum update
...

...
Transaction Summary
===================================================================
Install     18 Package(s)
Update     148 Package(s)
Remove       0 Package(s)

While I didn’t check all the packages in the list, XChat is definitely there:

xchat                   i386       1:2.8.4-15.fc9   fedora            1.3 M

So did Fedora just push out a release with old packages just to stay on schedule, and is actually releasing packages now hoping nobody would notice? (more…)

The highlight today of probably every Linux-related mailing list and IRC channel was the announcement of CVE-2008-0166, affecting OpenSSL libraries on Debian-based Linux distributions, including the popular Ubuntu.

According to the Debian Security Advisory, a change made to Debian’s OpenSSL package makes its random number generator predictable. Obviously this is less than desirable in a random number generator used for things like, say, all of your SSH keys.

The vulnerability has been present since September of 2006, and Debian strongly suggests throwing your old keys out completely:

It is strongly recommended that all cryptographic key material which has been generated by OpenSSL versions starting with 0.9.8c-1 on Debian systems is recreated from scratch.

Debian has now disabled public key authentication on their project servers until further notice, and are generating new keys for those servers and new certificates for db.debian.org.

So all you Debian and Ubuntu folks out there will probably want to do the same for your own keys and certificates. Note that this patch was never used by the upstream OpenSSL team nor by other distros like Fedora or RHEL (or CentOS), so they are not affected.

Ah, the perils of working in a shared, client environment. One client has us using a login that is not exclusive to us. I prefer using bash; the client is set to use zsh. This is not a problem in and of itself.

However, there is a section in the .profile that is causing me issues:

if [ -f /usr/bin/ksh ]; then
        /usr/bin/ksh -o vi
        exit
fi

So, “If ksh exists, run it with some options to edit history with vi-like commands”. Except what we really want is “If you’re using the ksh as a shell, . . . .”

So I added a modification, and now all is fine.

if [ -f /usr/bin/ksh ]; then
        if [ "$SHELL" = "/usr/bin/ksh" ]; then
                /usr/bin/ksh -o vi
                exit
        fi
fi

(not all my problems are MySQL related!)

I try to do a decent job of advocating for caring about good backups and business continuity strategies in my 7 Deadly Habits article.

But this one beats them all:

Just too funny and great not to share. Found via this reddit article, where there is a lively discussion underway.

In the era of consolidation, storage has not been left out. Different systems are made to share the same storage boxes, fiber-channel switches and networks. Inside a typical storage box, we have front-end and back-end controllers, cache, physical spindles shared amongst different applications, databases, backup destinations, and so on.

The impact of backup on normal database activity . . . batch processing in one database impacting transactional processing — these are two real life examples of the consequences of storage consolidation known to almost every DBA. Of course, it’s easy to suggest separating databases to different physical disks, but what about SAN box controllers and shared cache? And don’t forget about the cost factor and ubiquitous consolidation that forces storage administrators to pack as much data as possible into a single SAN or NAS storage device.

Some of our customers use hosting services — they outsource hardware hosting just like they outsource DBA work to Pythian. In such scenarios, hosting service providers usually have storage hardware shared amongst different customers to provide higher utilization and on-demand storage capacity at a lower cost.

(more…)