PSIgroup | Computing / SSHTunnel

An SSH tunnel can be used for many things but it's great for SSH access to a machine that is not visible for the wider internet. If there is an intermediary server that is accessible, then the target can establish a reverse tunnel to the server. A client can connect to the target, via the server.

Tunnel explicit port through a proxy

E.g. Unencrypted web traffic uses port 80. Firstly, an application that uses a forward tunnel is accessing a local-only website from a remote location. This can be used to access an intranet site, or control a http interface from elsewhere on the internet. It uses an externally reachable 'gateway' machine on the same network as the web site, e.g. proxymachine.

In a shell on your remote machine:

 ssh -L 1080:proxymachine:80 user@server_behind_firewall

and then in a browser (dedicated browser for tunnelling so you don't have to keep changing proxies) you set the proxy to localhost, port 1080, and have no exceptions. Alternatively, you may be able to browse directly to http://localhost:1080/ to access the site on the remote machine via proxymachine.

The -f and -N flags can be given to have SSH set up a port forward and become a background process, not actually running any command on the remote machine.

Dynamic SOCKS proxy

As above, a port can be opened on your local machine that tunnels to the remote machine. A dynamic port can be used as a SOCKS proxy for both encrypted and unencrypted TCP access, even on alternate ports. This is configured with the -D flag e.g:

  ssh -D 8080 user@gatewaymachine

You can then configure your browser to use this port as a SOCKS5 or SOCKS4 proxy.

Some browser extensions can dynamically pick a proxy based on URL. FoxyPAC is a Firefox addon that can match patterns for proxies. Proxy Helper is available from the chrome store. This requires a PAC file which is a javascipt function FindProxyForURL that returns the proxy type and URI for each request. An example PAC file for our systems is below:

function FindProxyForURL (url, host) {
  // Match explicit psi FQDNs
  if (dnsDomainIs(host, "uj.ac.za") && shExpMatch(host, 'psi-*.uj.ac.za') || (
      host == "bhubesi.uj.ac.za"
      || host == "didingwe.uj.ac.za"
      || host == "ukhozi.uj.ac.za"
      || host == "odin.uj.ac.za"
      )) {
    return 'SOCKS5 localhost:8080';
  }
  // Match 10.60.6.0/24 IPs
  if (isInNet(host, '10.60.6.0', '255.255.255.0')) {
    return 'SOCKS5 localhost:8080';
  }
  // Else don't use proxy
  return 'DIRECT';
}

In Proxy Helper, the PAC file can be specified in the extension options and the plugin mode set to `PAC Script":

FoxyPAC can have patterns specified in the extension options and the plugin mode set to "Proxy by patterns":

Reverse SSH tunnel

If you are at a remote location where the network is blocked by a firewall (iThemba, for example) it's impossible for someone at UJ, or anywhere else, to connect to your PC - which is the security that you want from a firewall, in general. But if you, for example, need someone from UJ to run a program on that PC - then it becomes a problem.

A way to circumvent this is to use SSH (Secure SHell) to make a "tunnel" from a target machine:

 ssh -R 11022:localhost:22 myserver

In this way port 11022 (or your choice of a random high port, >1024) on myserver (a server where it is possible to SSH in) will be temporarily be connected to port 22 (which is the standard SSH port) of your target PC. So your colleague at UJ can just login on to myserver, and do

 ssh -p 11022 target_user@localhost

and this will actually connect him to your remote PC at iThemba!

You can even SSH from another machine to an intermediary server and then on to the target in a single step:

 ssh -t myserver ssh -p 11022 target_user@localhost

HOWTO: Create a permanent ssh back-tunnel

This is for a permanent setup. For a once off case, the recipe above is sufficient.

Warning: if this looks complicated, it's because it is complicated! Actually, it's not that complicated, but it takes quite a few steps. Some of these steps could be avoided if OpenSSH on SL4 and SL5 supported the ExitOnForwardFailure option, but it doesn't; because of this, it's necessary to arrange an automatic reverse connection so that the script can check periodically that the tunnel is actually alive, and it's not just a plain ssh connection without the tunnel.

host1.example.com - incoming open
host2.example.com - incoming closed
Create a user

root@host1# useradd tunnel
root@host1# su - tunnel

Create ~/.ssh/ with the right permissions

tunnel@host1$ ssh localhost

Now generate the id-dsa key

tunnel@host1$ ssh-keygen -t dsa

Repeat steps 1-3 on host2

root@host2# useradd tunnel
root@host2# su - tunnel
tunnel@host2$ ssh localhost
tunnel@host2$ ssh-keygen -t dsa

Exchange keys

tunnel@host2$ cat .ssh/id-dsa.pub # then copy from one host
tunnel@host1$ cat >> .ssh/authorized_keys # and paste on the other (ctrl-D to end)
tunnel@host1$ chmod go-w .ssh/authorized_keys 
tunnel@host1$ cat .ssh/id-dsa.pub # then copy from one host
tunnel@host2$ cat >> .ssh/authorized_key # and paste on the other (ctrl-D to end)
tunnel@host2$ chmod go-w .ssh/authorized_keys

Connect from host2 to host1 to test and accept key fingerprints:

tunnel@host2$ ssh host1.example.com

Configure the tunnel with host2:~tunnel/.ssh/config:

Host host2_host1_tunnel
     Hostname host1.example.com
     User tunnell
     Compression no
     ForwardX11 no
     KeepAlive yes
     GSSAPIAuthentication no
     RemoteForward 10001 localhost:22
     #GatewayPorts yes # uncomment this to make this tunnel available to other hosts
     BatchMode yes
     #ExitOnForwardFailure yes # not implemented in OpenSSH 4.3
     ServerAliveInterval 3

Host host1_tunnel_test
     Hostname host1.example.com
     User tunnell
     Compression no
     ForwardX11 no
     BatchMode yes
     ConnectTimeout 30

Now run the tunnel manually from host2:

tunnel@host2$ ssh host2_host1_tunnel

On host1, Connect from host2 to host1 to test and accept key fingerprints:

tunnel@host2$ ssh host1.example.com

Append to host1:/etc/ssh/ssh_config:

Host host2
     Hostname localhost
     Port 10001
     HostKeyAlias host2_tunnel
     CheckHostIP no

This makes it easy for a user on host1 to use the tunnel. The HostKeyAlias and CheckHostIP options prevent problems with conflicting keys stored in .ssh/known_hosts, especially if you use multiple tunnels.

Now, to make this fully automatic, we need a script that runs the connection, and also checks that it actually works - because OpenSSH 4.3 only issues a warning if it cannot make the tunnel.
Script /usr/local/sbin/host1_tunnel.sh on host2:

#!/bin/bash
#set -x
#while true; do

T=host2_host1_tunnel
H1test=host1_tunnel_test
H2test=host2

LOG="logger -i -p authpriv.info -t tunnel "

if [ -z "$(pidof sshd)" ]; then
   $LOG "Waiting for sshd to come up"
   sleep 30
   exit 1
fi

SPID=$(ps xa | grep ${T} | grep ssh | cut -b-5)
if [ -z "$SPID" ]; then
   #echo "Starting tunnel"
   $LOG "Starting tunnel $T"
   ssh -f -N ${T}
   sleep 30 # give it time to come up
else
   if ! ssh $H1test "ssh $H2test -o ConnectTimeout=15 true 2>/dev/null" ; then
      $LOG "Killing dead tunnel $T at  $SPID"
      kill -9 $SPID
      sleep 10 # give it time to die
   else
     # All fine, rest 10 minutes
     sleep 600
   fi
fi
#done

To have the script running all the time, there are a few different solutions. My preferred one at the moment is to add the script to /etc/inittab on host2:

t1:345:respawn:/usr/local/sbin/host1_tunnel.sh

and have init reload it:

/sbin/telinit q

Notes:

No password is assigned to either tunnel@host1 or tunnel@host2. This means that these users can only connect using the SSH keys - which is good for security.

Known bugs:

The tunnel can fail for a while if the connection is temporarily dropped, and host1 still keeps the local sshd instance running and the tunnel port open, so that a new tunnel cannot use it, until the TCP socket dies of natural death. For a faster recovery, assuming the connection is up again, try the following:

host1$ sudo netstat -l -p --tcp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 localhost.localdomain:10001 *:*                         LISTEN      4915/sshd: tunnel

host1$ sudo kill 4915

Physics gateway machine

There is already a tunnel account on physics.uj.ac.za. Additional machines can tunnel in for remote access to specific services. First up, the machine running the service must be able to SSH to the tunnel account from its root account. Create an ssh key:

> sudo -i
# ssh-keygen -t ed25519
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_ed25519
Your public key has been saved in /root/.ssh/id_ed25519.pub
...
# cat /root/.ssh/id_ed25519.pub

Copy the public key into the authorized_keys file in the tunnel account on the physics.uj.ac.za host:

> sudo -i
# echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX root@Machine" >> ~tunnel/.ssh/authorized_keys

Test that the machine can now log into physics.uj.ac.za:

# sudo ssh tunnel@physics.uj.ac.za

If root can log in correctly as the tunnel user, the service can now be set up to maintain the tunnel. First install autossh:

# sudo apt install autossh

A systemd service can then be placed at /lib/systemd/system/tunnel@.service on the service machine with the following content:

[Unit]
Description=SSH Tunnel from %I port XXX22
Wants=network.target network-online.target
After=network.target network-online.target

[Service]
Environment="AUTOSSH_GATETIME=0"
ExecStart=/usr/bin/autossh -M0 -vv -o "ServerAliveInterval 10" -o "ServerAliveCountMax 3" -o "ExitOnForwardFailure yes" -N -R XXX22:localhost:22 tunnel@%i
Restart=on-failure
UMask=0066
StandardOutput=null
Restart=on-failure

[Install]
WantedBy=multi-user.target

Note to replace the XXX in both port numbers with an unused number on the physics.uj.ac.za machine. Also, if you are making a tunnel for a service other than SSH, change the localhost port from 22 to whatever is needed, and adjust the Description to match.

Update systemd, then start the new service:

# systemctl daemon-reload
# systemctl start tunnel@physics.uj.ac.za
# systemctl status tunnel@physics.uj.ac.za
  tunnel@physics.uj.ac.za.service - SSH Tunnel from physics.uj.ac.za port 11022
     Loaded: loaded (/lib/systemd/system/tunnel@.service; disabled; vendor preset: enabled)
     Active: active (running) ...
DATE MACHINE autossh[PID]: port set to 0, monitoring disabled
DATE MACHINE autossh[PID]: starting ssh (count 1)
DATE MACHINE autossh[PID]: ssh child pid is PID

If that all looks ok then also enable the service such that it start on system boot:

# systemctl enable tunnel@physics.uj.ac.za
Created symlink /etc/systemd/system/multi-user.target.wants/tunnel@physics.uj.ac.za.service &#8594; /lib/systemd/system/tunnel@.service.
# systemctl status tunnel@physics.uj.ac.za
  tunnel@physics.uj.ac.za.service - SSH Tunnel from physics.uj.ac.za port 11022
     Loaded: loaded (/lib/systemd/system/tunnel@.service; enabled; vendor preset: enabled)
     Active: active (running) ...
DATE MACHINE autossh[PID]: port set to 0, monitoring disabled
DATE MACHINE autossh[PID]: starting ssh (count 1)
DATE MACHINE autossh[PID]: ssh child pid is PID