-- Stanford Linear Accelerator Center

Secure Shell (SSH) at SLAC

UNIX at SLAC
Security
Updated: 13 April 2010
--

Contents

Secure Shell at SLAC
Using SSH
Between UNIX machines at SLAC
Connecting from OffSite
Problems and workaroundsNew Content
Reference

Secure Shell at SLAC

SLAC uses OPENssh software on all unix machines. Where possible we use the vendor supplied version, but for older machines, we use a locally compiled version. Since most SLAC home directories are in AFS, using ssh keys won't work well. We use GSSAPI authentication ( i.e. kerberos 5) and credential forwarding to allow the ssh daemon then runs a special command when you login to obtain an AFS token for you. 

Advantages of OPENssh.

  • If you have a kerberos 5 enabled version of ssh, you can do password-less login and still have an AFS token.
  • SSH consistently starts an AFS Process Authentication Group (PAG), providing for better protection and management of the AFS token.
  • SSH compresses the data in addition to encrypting it. This may result in a speedup on slow connections. 
  • SSH can pass other connections, such as X11, over the encrypted channel, making secure X connections possible.

Protocols in use at SLAC

There are two different SSH protocols, but only Protocol 2 supports GSSAPI login and forwarding of kerberos 5 credentials.  Due to a DOE mandate SSH V1 has been turned off at SLAC as of January 2008. SSH V1 is largely deprecated and all recent ssh clients should support V2.

The SFTP client and server are also installed at SLAC. The OpenSSH SFTP client uses SSH Protocol 2 by default.

Using SSH

Between UNIX machines at SLAC

SSH can be used either to initiate login sessions or copy files using scp. All taylored machines at SLAC automatically have their ssh host keys updated in the global known_hosts file every night. If you are at SLAC and see the warning about "unknown host key" when connecting to a SLAC machine, please contact unix-admin@slac.stanford.edu

Note: The latest version of ssh requires that you have a K5 kerberos ticket to use these commands without repeatedly typing your password. Most machines at SLAC have been setup to obtain a K5 ticket automatically when you first login, however you can always check if you have a ticket using the klist command. If you don't have a valid ticket use kinit to obtain one.

Login into a remote machine
ssh remote.slac.stanford.edu
Copy a file to a remote machine
scp local_file remote.slac.stanford.edu:remote_file

SSH tries several methods of authentication in sequence, and which one succeeds depends on your setup. The following sections describe the methods in the order that ssh attempts them. 

Connection using GSSAPI
If you have a .k5login file in your home directory that contains your full kerberos principal ( userid@SLAC.STANFORD.EDU) and you have an ssh client that is capable of transferring the kerberos ticket on the machine your are logging in from, then ssh will allow you to login without a password and automatically obtain an AFS token for your login session. This is by far the preferred method of authentication and should be used whenever possible. For more details see this page.
Connection using RSA or DSA
SSH gives you the ability to generate your own public/private key pair and use that to authenticate your logins. There are some drawbacks and disadvantages. First, your private key must either be locked with a passphrase that you have to enter any time you log in, or it must be stored in a very secure machine. A private key without a passphrase is like storing your password on disk for anyone to read; anyone possessing it can log in as you. Second, RSA authentication does not get you an AFS token when you log in. ( only password or GSSAPI authentication will get you a new token.) If you do generate an RSA key, either protect it with a passphrase, or store it on a local hard disk or a floppy disk that you carry with you. Never store a private key that isn't protected by a passphrase in an NFS-mountable directory. For Protocol 2 you should use a DSA keypair created as follows.
ssh-keygen -t dsa -C "USERID@slac.stanford.edu"

Note: We strongly discourage the use of ssh keypairs for login to SLAC machines. Many things will not work correctly if you do not obtain an AFS token during the login process.

Connection using your password
If neither of the other two authentication methods works, ssh will prompt you for your Kerberos password and will try to log you on and get an AFS token for you. It's important to realize that ssh is encrypting your entire session, so your password remains secure even though it is sent across the network. However, with the advent of keyboard sniffing viruses and other malware, the less you type your password the better.

Between a SLAC machine and a non-SLAC machine

This case works much the same way as between SLAC machines, except that you have to concern yourself a bit with host keys, which are managed automatically for you on SLAC machines. Each host has a private key and a public key. In this explanation, we will call the host you are connecting from the client machine, and the host you are connecting to the servermachine. When you first connect to a server that ssh on your client does not know about, it will ask whether you want to accept the public key of that machine. It will store that key in a file in your home directory on your client named ~/.ssh/known_hosts. Every connection after that will check the public key of the server, and will give you loud warnings if it is ever different. This protects you from hacker attacks in which another machine impersonates the trusted server machine to which you are trying to connect. The dialog looks like this:

> ssh flora.slac.stanford.edu
Host key not found from the list of known hosts.
Are you sure you want to continue connecting (yes/no)? yes
Host 'flora.slac.stanford.edu' added to the list of known hosts.

If you get the wrong key from the remote host, you get this dialog:

> ssh flora.slac.stanford.edu
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@        WARNING: HOST IDENTIFICATION HAS CHANGED!        @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the host key has just been changed.
Please contact your system administrator.
Add correct host key in /u/sf/boeheim/.ssh/known_hosts to get rid of this message.
Agent forwarding is disabled to avoid attacks by corrupted servers.
X11 forwarding is disabled to avoid attacks by corrupted servers.
Are you sure you want to continue connecting (yes/no)? No

In fact, you will often get exactly this warning if you connect to pooled machines such as rhel5-64, flora, or rhel6-64, since you will get different machines each time, with different host keys. The simplest solution to this is to obtain the host keys of all the machines in the pool from SLAC with this command, run from the client machine. These keys are available via anonymous ftp from the server ftp.slac.stanford.edu, in the directory /admin/:

ftp://ftp.slac.stanford.edu/admin/known_hosts

Copy this file to your client machine and merge it into your ~/.ssh/known_hosts there, replacing any existing keys for the corresponding SLAC hosts. From time to time you may need to repeat this process, for example, when SCS adds new machines to any of the pools.

When connecting from a client machine at SLAC to a remote server at another site, you will get a similar dialog. In this case you will be saving the public keys of the servers that you connect to in the file ~/.ssh/known_hosts in your home directory at SLAC. 
 

OpenSSH and Changed Host Keys

Note that OpenSSH will not permit password authentication if the remote host's public key has changed.  On the other hand, if a user is authenticating via an RSA key, there is no problem connecting to a host with a changed host key, so long as "StrictHostKeyChecking no" is set in the system-wide ssh_config file, or used on the command line via the -o option, or set in the user's personal ~/.ssh/config file.  

For example:

ksa@osiris $ ssh elaine.stanford.edu
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The RSA1 host key for elaine.stanford.edu has changed,
and the key for the according IP address 171.64.15.106
is unknown. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA1 host key has just been changed.
The fingerprint for the RSA1 key sent by the remote host is
26:b7:69:94:1b:4e:65:7c:6d:4a:11:ed:ed:fd:04:77.
Please contact your system administrator.
Add correct host key in /u/sf/ksa/.ssh/known_hosts to get rid of this message.
Offending key in /u/sf/ksa/.ssh/known_hosts:1
Password authentication is disabled to avoid trojan horses.
Agent forwarding is disabled to avoid trojan horses.
X11 forwarding is disabled to avoid trojan horses.
Permission denied.
Exit 255.

So, to still be able to connect to a host with a changed host key using an OpenSSH client, users will have to:

  1. Use RSA authentication instead of password authentication
  2. Remove bad entries from their personal known_hosts file
  3. And it would be a good idea to contact the sysadmin to verify why the host key has changed.

In the above example, elaine.stanford.edu is a pool of hosts (much like rhel6-64, flora, and tersk at SLAC). Therefore, the correct thing to do is put all the host keys for the elaine pool in the user's ~/.ssh/known_hosts file.

If the bad host key is in the system's global known_hosts file, then the user needs to put the good host key in his/her personal known_hosts file, which gets searched first. Note that the option 'StrictHostKeyChecking ask' means the user will get prompted before connecting to a new, previously unknown host.

Connecting w/o a Password from Remote Sites

This requires an ssh client that is capable of using GSSAPI authentication with kerberos 5 and forwarding the credentials. Fortunately most of the major Linux versions come with this capability and all recent versions of OS X also come with this software as part of the default installation. For more details on how to set this up see Using GSSAPI authentication.

Connecting from a PC

There are commercial and open source clients available for Windows PCs. Secure CRT is an excellent choice, but is not free. If you have a sunetid (i.e. SLAC employee), you can get it for free from the Essential Stanford Software distribution.

In Mac OS X, the open source OpenSSH client is included with the basic OS install, it also includes the klist and kinit commands used above.

Forwarding other services with ssh

SSH can forward other TCP services over the encrypted connection. Examples of such services would be FTP, POP, IMAP, and X-Windows. This keeps the passwords that these services forward over the network from being visible to hackers who may be watching the network traffic. These services have no encryption of their own built in, and need the protection of an external protocol. This forwarding is often referred to as tunneling, because the TCP traffic is sent through an encrypted tunnel that shields it from view.

The forwarding of X-Windows traffic is automatic with ssh, unless you explicitly disable it. To start up an X command on a remote host, start it with an ssh command like this:

ssh -f outpost.podunk.edu xterm
This will start an ssh session, which will direct the traffic to and from the xterm along the encrypted tunnel. X Magic Cookie authentication is automatically handled for you. This method is the only secure way of opening an X session across SLAC's firewall, which blocks incoming X traffic (incoming is when the display screen is at SLAC and the program is running offsite). 

FTP is more difficult to forward, since an FTP session is made up of two connections, one from the client to the server to send the login information, password, and command, and another from the server back to the client to transfer the data. It has proven difficult to give a formula for making this work with arbitrary FTP clients, as most common clients and servers have added protections against hackers redirecting sessions to unintended destinations. Your best bet for transferring data between machines is the scp command. If you must use FTP, first see if you can use anonymous FTP servers for making data available and anonymous FTP drop-boxes for transferring data in. If you have to use authenticated FTP (with a username and password), you can try using ssh to forward port 21 to encrypt only the control channel, or you can see if your client and server support passive mode transfers.

Problems and work-aroundsNew Content

Here are some current problems you might encounter when using SSH at SLAC, along with suggested work-arounds.

"Hangs" when logging off

If you login to a host via SSH, start a process in the background, and then try to logoff leaving the background process running, your logout will hang if the background process has left any of its standard IO file descriptors open. In this case you get no warning message, the logoff simply hangs.

In either case, you can force a disconnect by typing the SSH escape sequence, "~.". (all SSH escape sequence must be entered immediately following a CR). This will break any open X connections and kill the background process. If you need to leave the background process running, you should do something like this (for csh/tcsh users):

     some-command </dev/null >&/dev/null &
     exit
The syntax for Bourne/Korn/BASH shell users is slightly different:
     some-command </dev/null >/dev/null 2>&1 &
     exit
To launch a background process on another host from a script, you could do something like this (assuming your default shell is csh or tcsh):
     ssh -x some-host some-command '</dev/null >&/dev/null &'

SSH and SLAC

  • Shared accounts will need a slighty different setup, rather than using the ~/.ssh/authorized_keys file to control access, you will want to use the ~/.k5login file to list the authorized users of the account. The ~/.ssh/authorized_keys method will still continue to work, but you will have to kinit after login to obtain an AFS token. SCCS has created an automated process to allow groups to manage the .k5login file effectively. See this page for more information.

  • X11 forwarding works in a more secure way than may cause problems with older or non-comforming X11 programs. If you notice strange behaviour try using

    ssh -Y

    This reverts ssh to the older X11 forwarding.

  • Only your current k5 tgt will be forwarded and kerberos is generally only setup to deal with one realm/tgt per file. If you have been getting afs tokens in multiple realms and using ssh to forward all those tokens that will no longer work. You can use the KRB5CCNAME environment variable to get tgt's for multiple realms

    env KRB5CCNAME=/tmp/myid_remoterealm kinit myid@REMOTE_REALM

    will get an alternate set of credentials.

  • Some handy ssh tricks

Reference

Man pages for the ssh commands

There are detailed manual pages for each of the primary ssh commands. Use the man command on the UNIX system, or click on the link below to view them. 

OpenSSH Manual Pages

SSH information elsewhere


OpenSSH home page
OpenSSH FAQ
SLAC SSH FAQ
Using GSSAPI Authentication
Handy SSH tricks
SSH for shared accounts

Owner: Booker C. Bense