I was contacted by a colleague about a problem he was having. “I’m trying to set up something simple which I’ve done millions of times, but it’s not working. I might be missing something obvious.”
The issue was that the SSH public key authentication didn’t work. The environment was running a virtualized Oracle Enterprise Linux 6.4 operating system (similar to Red Hat Enterprise Linux RHEL or Centos 6.) We’ll call this box Badboy for the purpose of this post.
I logged onto Badboy and attempted to do it myself, following the basic steps to set up public key authentication on Linux:
[knecht@random-client ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/knecht/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/knecht/.ssh/id_rsa.
Your public key has been saved in /home/knecht/.ssh/id_rsa.pub.
The key fingerprint is:
The key's randomart image is:
+–[ RSA 2048]—-+
| . |
| + o |
| . * + . . |
| . = S . ..+ |
| *o…+. o |
| E o…. + .|
| . . . .|
| . |
I then copied that public key file to Badboy and added the key to the
Let’s verify that the public key authentication has not been disabled in
[root@client ~]# grep Pubkey /etc/ssh/sshd_config
The value isn’t set explicitly, so the default setting comes into play which is
I then switched back to my random-client and attempted to log in using SSH with the public key.
[knecht@random-client ~]$ ssh knecht@badboy
Hmm, it’s asking for a password. That’s not what you would expect. Let’s disable password authentication entirely in the client:
[knecht@random-client ~]$ ssh -o "PasswordAuthentication no" knecht@badboy
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
This box was really not behaving nicely. Badboy it is then.
I checked the logs to see if anything shows up by doing a
tail -f /var/log/secure while logging on from a second session, trying again with password authentication disabled.
The only entry that showed up was:
Nov 21 20:14:21 badboy sshd: Connection closed by xx.xx.xxx.xxx
That’s helpful. Move on.
Hoping to get a little bit more insight, I increased the logging verbosity of sshd, by changing
Retrying the previous exercise, we now have a wee bit more information:
Nov 21 20:21:59 badboy sshd: Failed publickey for knecht from xx.xx.xxx.xxx port 36189 ssh2
Nov 21 20:21:59 badboy sshd: Connection closed by xx.xx.xxx.xxx
Well, you’re not very talkative Mr. Badboy, are you? I went again and triple-checked all of the possible options, read the main pages, and couldn’t determine what was wrong with Badboy’s configuration. As my colleague mentioned, it’s a simple exercise. We’ve all done it countless times… So what was the problem now?
It dawned on me that I had a similar experience in the past, when things just wouldn’t work without any clear reason. The new suspect was now Mr. Badboy’s big brother, SELinux.
[root@badboy ~]# sestatus
SELinux status: enabled
SELinuxfs mount: /selinux
Current mode: enforcing
Mode from config file: enforcing
Policy version: 26
Policy from config file: targeted
Well, there you have it. A quick peek in
/var/log/audit/audit.log showed several actions being denied during an attempted connection.
How to configure SELinux to get this to work is beyond the scope of this document. In our environment, we are not using SELinux so we disabled it by setting
/etc/selinux/config and rebooted the system.
As soon as it was back up:
[knecht@random-client ~]# ssh knecht@badboy
Interested in working with Stefan? Schedule a tech call.