Why was SSHD refusing my key?

Nov 22, 2013 / By Stefan Knecht

Tags: , , , , , , , , , , , , ,

I was contacted by a colleague about a problem he was having. “I’m trying to set up something simple which I’ve done millions of times, but it’s not working. I might be missing something obvious.”

The issue was that the SSH public key authentication didn’t work. The environment was running a virtualized Oracle Enterprise Linux 6.4 operating system (similar to Red Hat Enterprise Linux RHEL or Centos 6.) We’ll call this box Badboy for the purpose of this post.

I logged onto Badboy and attempted to do it myself, following the basic steps to set up public key authentication on Linux:


[knecht@random-client ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/knecht/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/knecht/.ssh/id_rsa.
Your public key has been saved in /home/knecht/.ssh/id_rsa.pub.
The key fingerprint is:
0c:32:d6:9e:4f:82:7d:f6:ff:c7:f3:0b:c6:a4:88:29 knecht@random-client
The key's randomart image is:

+–[ RSA 2048]—-+
| |
| . |
| + o |
| . * + . . |
| . = S . ..+ |
| *o…+. o |
| E o…. + .|
| . . . .|
| . |
+—————–+
I then copied that public key file to Badboy and added the key to the ~/.ssh/authorized_keys file.

Let’s verify that the public key authentication has not been disabled in sshd_config:

[root@client ~]# grep Pubkey /etc/ssh/sshd_config
#PubkeyAuthentication yes

The value isn’t set explicitly, so the default setting comes into play which is yes.

I then switched back to my random-client and attempted to log in using SSH with the public key.


[knecht@random-client ~]$ ssh knecht@badboy
knecht@badboy's password:

Hmm, it’s asking for a password. That’s not what you would expect. Let’s disable password authentication entirely in the client:


[knecht@random-client ~]$ ssh -o "PasswordAuthentication no" knecht@badboy
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
[knecht@random-client ~]$

Interesting.

This box was really not behaving nicely. Badboy it is then.

I checked the logs to see if anything shows up by doing a tail -f /var/log/secure while logging on from a second session, trying again with password authentication disabled.

The only entry that showed up was:


Nov 21 20:14:21 badboy sshd[4299]: Connection closed by xx.xx.xxx.xxx

That’s helpful. Move on.

Hoping to get a little bit more insight, I increased the logging verbosity of sshd, by changing LogLevel in sshd_config</code:

LogLevel VERBOSE

Retrying the previous exercise, we now have a wee bit more information:


Nov 21 20:21:59 badboy sshd[4980]: Failed publickey for knecht from xx.xx.xxx.xxx port 36189 ssh2
Nov 21 20:21:59 badboy sshd[4981]: Connection closed by xx.xx.xxx.xxx

Well, you’re not very talkative Mr. Badboy, are you? I went again and triple-checked all of the possible options, read the main pages, and couldn’t determine what was wrong with Badboy’s configuration. As my colleague mentioned, it’s a simple exercise. We’ve all done it countless times… So what was the problem now?

It dawned on me that I had a similar experience in the past, when things just wouldn’t work without any clear reason. The new suspect was now Mr. Badboy’s big brother, SELinux.


[root@badboy ~]# sestatus
SELinux status: enabled
SELinuxfs mount: /selinux
Current mode: enforcing
Mode from config file: enforcing
Policy version: 26
Policy from config file: targeted


Well, there you have it. A quick peek in /var/log/audit/audit.log showed several actions being denied during an attempted connection.

How to configure SELinux to get this to work is beyond the scope of this document. In our environment, we are not using SELinux so we disabled it by setting SELINUX=disabled
in /etc/selinux/config and rebooted the system.

As soon as it was back up:


[knecht@random-client ~]# ssh knecht@badboy
[knecht@badboy ~]#

Good boy.

7 Responses to “Why was SSHD refusing my key?”

  • Stoyan says:

    You may want to check the manpages of getfacl, setfacl and setenforce.
    I also see, the knecht user, uses UID 0 – logging with root account is something, strongly advised against (Please, accept my apologies, if the hash (#) was set up, using PS1 :) )
    But in general, I agree – SELinux is a drag, most of the times.
    My best regards.

    • Stefan Knecht says:

      You’re absolutely right regarding root logins – however in this particular case it wasn’t a production environment (where direct logins as root are usually disabled from the start).

      SELinux certainly has its uses, I do find it to be a very good thing in high-security environments. However, in this particular case it wasn’t something I would have expected.

  • Leon Boot says:

    Thanks for this post; I was on the right track all the way up to changing the SELINUX variable to ‘disabled’ in /etc/selinux/config but I hadn’t rebooted yet. I believe this step wasn’t necessary in earlier versions of [fill in your favourite RHEL flavoured distro]?

  • Stoyan says:

    Something, I totally forgot – if you decide, to disable SELinux, you don’t need to reboot the system.
    Do ‘/sbin/setenforce 0′
    Have a great weekend :)
    Stoyan

  • Jan says:

    Thanks a lot for this write-up. I had already wasted precious time trying to figure out why it would work when running /usr/sbin/sshd -d but not when running as service. SELINUX definitely was the culprit. Need to figure out how to use public keys WITH SELINUX enabled …

  • Stoyan says:

    Happy new 2014.
    SELinux is a bit tricky – here is 1 possible way to go:
    semanage port -l
    # this is a list
    semanage port -a -t ssh_port_t -p tcp 22
    # this will add tcp 22.
    service sshd restart

    To delete (block) a port from the group:
    semanage port -d -t ssh_port_t -p tcp 22

    One thing – the above is more of a POC – like I said – SELinux depends on many things.
    Hope someone finds this useful :)
    My best

Leave a Reply

  • (will not be published)

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>