Chris' SSH Notes

SSH is a public key cryptographic system. This means that it does not use a symmetrical "shared secret", i.e. the password I encode with may not be the password you decode with. Instead it generates a key pair, one private and one public. The private key is very important and must be very closely guarded. To let someone have access to it is to improve their odds of breaking into what it protects substantially. The public key is not sensitive information and that can be freely distributed without concern.

Let’s say you have a host called house and another called office. You want to work from home which entails logging into office from house. On house generate a key pair. Create a file on office called ~/.ssh/authorized_keys which contains the public key from house. This file can contain several public keys, one per line. It’s fine to edit it manually by cutting and pasting or any way you can.

Note the permissions section below to sort out any permissions issues.

Assuming your key is named housekey, and the contents of housekey.pub are in office:~/.ssh/authorized_keys, you should be able to log into office from house like this:

If you ever work with a cluster, it is common that each node on the cluster will have a common shared file server where a user’s home directory is stored. This means that a file changed on node n12 will also be changed on node n17. Often it is useful to set up a key pair with no password so that you can easily control things on any node of the cluster. Since each node is probably on a local network (i.e. not available to the general internet) the security issues are less problematic. Also, if any node is compromised, with a shared file system all nodes are then compromised regardless of what key pairs are lying around.

To set this up is relatively easy since your home and office hosts as used in the previous examples are basically the same machine as far as setup goes. Perhaps the easiest way is to just run:

And then accept defaults and when it asks for a password, just hit enter (it will ask for confirmation - hit enter again). Then it will create the key pair in ~/.ssh. You can ls -l ~/.ssh to have a look. Now that you have a key pair you can use, you have to tell your account (which is replicated on all the nodes) that you will accept this key pair for log in attempts. A good way to do that would be:

If you had an authorized_keys file you have now just appended the contents of your new public key to it (this example assumes that the new public key was ~/.ssh/id_rsa.pub, but it could be ~/.ssh/id_dsa.pub or some other name you gave it).

Note the permissions section below to sort out any permissions issues.

Once the keys are set up and have the correct permissions, you should be able to log into another machine without a password. Just

If node22 has the same home directory source, it should find your authorized_keys file there (even though you set it up "here") and it should log in nicely. If there are problems, try using the -v option. This always is a helpful first step to diagnosing SSH key problems.

Often there are problems with permissions. Permissions are important because you don’t want all this security only to have some one else log into the system and just read your ssh set up and learn what strategies would best be used to attack your account. This is why permissions can be fussy.

Note that the authorized_keys file should have no permissions for group or other or it will not work. Do this to make sure:

You also have to have private keys be private for SSH to take them seriously:

The public keys can be any way you like; they’re public after all:

I also think it’s a good idea to do the following:

This limits everybody but you and the SSH server from seeing how your security is configured.

If your private key is encrypted with a passphrase (a very good idea), here is how you change it:

Here is how to limit what a key can do by origin and by function:

Just manually edit your authorized_keys file and include the keys you want to authorize with any restriction prefixes.

Note that the command= directive won’t block a different command, but rather it will just run the command specified in the authorized_keys file. So if you have this key in host alpha:.ssh/authorized_keys:

And do this from beta:

You’ll get the results of cal. In other words, instead of simply not working, it works differently than you explicitly commanded it to.

How do you know what command is really being run remotely? Have SSH monitor it and report it back to you. To do this use a program like this:

This is useful if you’re trying to set up a key that only allows a specific rsync job. Using test_ssh_cmd, you can run your rsync command and see that it really runs on the sshd server like this:

Then you can clip that out and put it verbatim into a command="" specifier and it should work. To otherwise just guess this can be difficult.

Private keys that are used interactively should always be protected with a passphrase. However, it’s reasonable to decrypt the passphrase and leave it available in memory for the duration of a session you might want to use it. This allows you to use the key without entering a passphrase every time. This is accomplished with a key agent. This is a special program that SSH knows how to consult with to see if any of the encrypted keys it needs to use have been decrypted recently and loaded in memory. If so, it just uses it again.

I don’t know the details, but it looks like by default Ubuntu (and probably other distributions' display managers) uses a Gnome key manager that seems pretty easy to use. Basically when you need a private key decrypted, a window pops up and offers you the chance to load it into its manager. Then subsequent usage of this key will use it from the key manager’s copy and not bug you for the decryption passphrase.

Sometimes you don’t have a GUI and a fancy integrated keyring manager won’t be possible. If you’re trying to get a key agent working with text console logins you have to resort to the standard key agent. The classic key agent is called ssh-agent. It’s kind of an unusual program. It basically needs to be the top level parent process of all the processes that might be running that would require its services. This can be a pain if you’re already running a shell that requires it and the agent wasn’t set up to be a parent process. So you have to plan in advance. This is done by running it and then running your shell as a sub process.

The most basic way to have a process (and its children) know about the key-agent is to make that process a subprocess of the agent. Here are some likely examples of how you’d use it in a simple way:

Here’s an example I use:

This alias is useful if you often use a computer without graphics, but when you do want graphics, this starts it up and makes the entire X Windows session a sub process of the ssh-agent. This means that whatever xterm you might have open, all of it is in touch with the key-agent. (Note that this alias also cleanly shuts down the tty console you were using which I think is a nice security feature so that people can’t Ctrl-Alt-F1 into the shell you started X from when you’ve locked the screen.)

Basically the agent works by exporting some important ssh-agent environment variables to the subprocess. You can also just have ssh-agent generate the shell commands to establish the current shell as an environment that knows about the agent. To do this, run (only one of these):

or

After this is done the environment variables should be in place for ssh clients to know how to use the agent.

These are untested, but seem reasonable for getting a key agent environment set up automatically from a .bashrc (in Bash):

Having this in bash allows the new shells to find the agent. To start the agent the first time, use something like this:

You can also install the program "keychain" which manages a lot of this mess. Apparently, you run the program:

And it creates files in a directory .keychain:

Now you just have to source one of the files in ~/.keychain:

or

Now when you log in from somewhere else, you can use the same key agent by sourcing this keychain script.

To stop a keychain spawned agent (and get rid of the .keychain scripts):

Having an agent keep your keys in memory is cool, but sometimes you need something fancier. You’re on hostA and you need to log in to hostC from hostB. This could happen, for example, if you’re at home on hostA and hostB is the secure gateway that lets you in through the firewall to hostC. I also use this setup to have a system administration machine (hostB) at work that controls various machines (hostC) from home (hostA).

The first thing to check is that hostB allows agent forwarding.

If not, reset this to yes and restart you ssh server. Once this is enabled, hostB pretends to have an agent running even when it doesn’t. When you ask the agent for keys, it basically forwards this request over the already established connection to hostA. Apparently if hostB is messy with untrusted activity, it might be best not to do this since an attacker can hijack all your loaded keys if there is a compromise. Otherwise I think it’s pretty safe.

Also your client on hostA needs to be cool about giving the agent information to your connections that ask for it. You need to set this on hostA:

Note that you can do this per host so that only certain hosts get the forwarded keys stored in the agent. Shown above is how to do it for all hosts. You can also do it on the command line using:

To see if things have a shot at working you can use this command to see what keys are loaded on hostA:

Use -L to see the full keys. If hostA shows you some keys, then you know your agent is working at least. Next run that same command on hostB. You should see the exact same keys available. If in doubt, you can delete a key from the list back on hostA and check to see if that did the same thing on hostB. Use this to remove a key from the agent’s control:

Once you have keys loaded in an agent, real or forwarded from somewhere else, you should be able to proceed as if it were a simple A to B connection.

Now for a very unusual but very irritating problem. Let’s say that for some crazy reason, you want to have a home directory that is not writable. This means that the normal place (~/.ssh/known_hosts) can’t accumulate host keys.

There are some things that can be tried. If you are the admin for a bunch of machines (maybe a compute cluster) you can think about adding all the nodes' host keys to /etc/ssh/ssh_known_hosts which contains a system wide definition of known host keys. This relieves users of having to verify them.

Another possibility that might be even easier is this option:

That’s a lot to type and I’ve had trouble with it believing my intentions sometimes.

Another trick is to collect the host keys in a non-typical place and put them in the right place some other way (elsewhere maybe). To do that use this option:

Sometimes you’re trying to sort things out with the known host keys and you find something like this:

This is a hashed known host key. The idea here is that if an attacker compromises an account, the first thing they will attempt is to look at your known hosts file for a record of exactly where else you like to log in. Then they’ll try those credentials that got them into this account on those accounts. This is how SSH viruses propagate. By hashing the host key, you have enough information for verifying a host, but not enough to propagate an attack to other machines SSH is known to have contacted. That said, this is a huge pain to deal with in many cases. The way to turn off (or on) hashed host keys is to set the option:

To fix this permanently, add this option to /etc/ssh/ssh_config. It’s off by default (if not configured) though configured on by default in some distributions.

Also take a look at the -F option of ssh-keygen to deal with unruly hashed keys. If the idea of hashed host keys sounds great to you you can convert a plain host key file with ssh-keygen -H filename.