Resolved: “Permission denied (publickey).” when trying to access via ssh an Ubuntu Server

So, this was a very tricky one and I had to turn off mandatory key authentication for the longest time because I just couldn’t figure it out. Now, keep in mind there are several reasons why this might be the case, so I will try to describe the symptoms, so you can determine if this post is for your or not. This seems to be a bit obscure as it took me a while to find and without knowing the cause it is hard to hit the right keyword search. So, let’s get this started:

Symptoms:

You’ve set up your ssh key correctly on the server and you tried to ssh into it with no problem using your public/private key. However, you restart your server and you get the dreaded message:

“Permission denied (publickey).”

So, you login manually into the server and find everything looks correct. You try once again and voilà! You can log in using your public/private key. So, what’ gives? You restart your server again or the ssh service and once again “Permission denied (publickey).” So, looking at the symptoms you can deduce that something is not working right but when you log in directly into the server (physically / console) it works. Not a whole lot of clues but after lots of persistence I was able to find out why:

Encrypted Home Drive

You’ve got to be kidding me, right? So, it never occurred to me two things:

1. This behavior only happens on servers I installed… and therefore chose to encrypt my home directory (Cloud servers don’t by default). I guess I realized the first part but not that a fundamental difference in the installation was the choice of an encrypted home drive
2. The ssh service cannot access the encrypted home folder (where they public key is stored under /home/<username>/.ssh/authorized_keys or like in the config file: %h/.ssh/authorized_keys)

 

So, lots of hours and frustration when the cause was rather something so simple… I am even surprised this is a thing and not handled in a better way (or warn about) when choosing an encrypted home drive. So, what’s the solution you say? Well, you get to choose between decrypting your home drive or storing your ssh keys elsewhere. I chose option number two and here is my walkthrough:

How to store your ssh public key in a different directory