Bypassing CG-NAT

Last week I dumped Vodafone Internet + TV bundle for a dirt cheap DIGI unlimited 5G data plan for my home. I have zero interest on TV… so it felt kind of dumb to be paying for something that I really don’t use or care.

Everything working fine except that I couldn’t access my cloud server from outside my home network… I traced the problem to DIGI using CG-NAT. Carrier Grade Network Address Translation is a technology that shares the same public IP address between multiple customers. The upside: it saves IPV4 addresses, the downside: it doesn’t allow for inbound connections.

CG-NAT blocks outside connections because many devices share one public internet address. The router managing this can’t tell which device an outside request is for, so it blocks them to keep things secure and organized.

A solution is top set up a reverse SSH tunnel. You establish a secure connection to an outside device with a public IP address that forwards internet requests through it to your CGNATed device via the encrypted tunnel.

Great news this software tunnel is very easy to set up. First check for port availability on the outside device. Let’s say you want to use port 8080. Check if the port is available

lsof -i :8080

or

netstat -tuln | grep 8080

make sure nothing is using the port you will assign to your service.

Set up ssh keys login with no password between your home network device and the outside device. You should be able to login like

ssh outsidedevice

remember the -v option can be very helpful debugging ssh troubles. On the outside server adjust GatewayPorts in sshd configuration:

sudo nano /etc/ssh/sshd_config
GatewayPorts clientspecified

and restart the sshd service

sudo systemctl restart sshd

now actually dig the tunnel. From your home device connect to the outside server, replace 192.168.1.100 with your internal device IP and adjust the ports of your service and bind address..

ssh -R bind_address:8080:192.168.1.100:8080 outsideserver -v

For simplicity sake, probably you want the same port on both machines. But if for some reason you want different ports the syntax is:

ssh -R outside_ip:outside_port:inside_ip:inside_port outside -v

If for some reason you want to bind to all addresses available:

ssh -R '*:outside_port:inside_ip:inside_port' outside -v

Or simply change the GatewayPorts directive in /etc/ssh/sshd_config to yes.

Now you should be able to access from anywhere via outside, lets say http://outside:8080 the service running inside your home network at lets say http://192.168.1.100:8080 bypassing CGNAT.

If your tunnel silently freezes up when data is transmitted trough it, please try lowering the MTU in the local network device. I was having this kind of problem and lowering the interface MTU from 1500 to 1400 solved it.

ifconfig eth0 mtu 1400

replace eth0 with your interface, and if needed to stabilize your ssh tunnel connection, don’t forget to update /etc/network/interfaces to make this change permanent across reboots.

At this point probably you are happy with the setup and want to auto magically connect the tunnel at startup and re-connect in case of unexpected tunnel closing. The tool for that is autossh. Install it:

sudo apt install autossh

test it manually first

autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -R 8081:192.168.1.100:8081 outsideserver -v

If all good make a systemd script to control it

sudo nano /etc/systemd/system/autossh-tunnel.service

adjust according

[Unit]
Description=AutoSSH tunnel to Outside Server
After=network-online.target

[Service]
User=root
Environment="AUTOSSH_GATETIME=0"
ExecStart=/usr/bin/autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -o ExitOnForwardFailure=yes -NR 8080:192.168.1.100:8080 outside
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Then reload the service daemon, enable your autossh tunnel service, start it, and confirm its status.

sudo systemctl daemon-reload
sudo systemctl enable autossh-tunnel.service
sudo systemctl start autossh-tunnel.service
sudo systemctl status autossh-tunnel.service

Boom, there you go your home network service again accessible trough the internet.

Setting up SSH access with server keys (no password) in FreeBSD

You want to login from user@host-a to user@host-b automatically (with no password prompt).

Install ssh-copy-id in the host-a. This little handy script takes care of all the dirty details from your back.

cd /usr/ports/security/ssh-copy-id
make install clean

then run (still in host-a)

ssh-copy-id user@host-b

If you get the error “ERROR: No identities found” then you must generate your keys first and then re-run ssh-copy-id. To generate a pair of private/public SSH keys you must issue:

mkdir ~/.ssh
chmod 700 ~/.ssh
ssh-keygen -t rsa

You can leave the pass-phrase empty for automatic logins (no password prompt) or the more secure but less practical password prompt each and every time that you use the newly generated keys.

Next login from host-a to user@host-b will be made with SSH keys.

The dirty details

  • the user@host-a public key, usually ~/.ssh/id_rsa.pub is copied to host-b user/.ssh/authorized_keys
  • the host-b public host key (/etc/ssh/ssh_host_rsa_key.pub) is copied to host-a known hosts

Securing SSH with SSHGuard

SSHGuardIf you have a remote server running some flavor of Unix or Linux 99% chances that you use SSH. The best security practice is to use an access key with password and disable password access altogether. But you end up loosing some flexibility (for some customers Putty is this utterly complex piece of software, imagine them playing with SSH keys…).

The best you can do is to enforce a better user password policy, but even so, as every password service it’s at mercy of brute force attacks. These attacks consume precious clock cycles and worst case scenario they can break a password and gain access to the system.

So, here comes SSHGuard to our rescue. It’s a pretty neat piece of software that is highly flexible and customizable to ones system, needs and paranoia level. On top of that is maintenance free and very easy to setup.

I’m using FreeBSD and the venerable (yet, very capable) IPFW firewall. The choice of the firewall is simply because it’s the one that i am more pro-efficient with.

First thing is to enable IPFW on your system. Open /etc/rc.conf and add these lines

firewall_enable="YES"
firewall_type="open"

actually this setup is only to bring IPFW up, it doesn’t filter anything, all the traffic is passed trough. But if you forget the firewall_type=”open” rule and start the firewall you will be lock out, because the default is no traffic allowed… (and you win a drive to the data-center or some kind of remote rescue shell procedure).

Start IPFW

/etc/rc.d/ipfw start

and check that is running

ipfw show

Now, you are ready to install SSHGuard itself, very easy task

cd /usr/ports/security/sshguard-ipfw
make install clean

and enable it in /etc/rc.conf

sshguard_enable="YES"

Ready? Start it

/usr/local/etc/rc.d/sshguard start

Still, there is a final thing to take care. SSHGuard uses syslogd to monitor incoming (failed) logins. So, you must edit /etc/syslog.conf and uncomment (or add if it’s not there) the line that the SSHGuard port added.

auth.info;authpriv.info     |exec /usr/local/sbin/sshguard

And restart syslogd

/etc/rc.d/syslogd restart

And now your SSH service should be bullet proof to brute force attacks. Keep safe!

UPDATE 2014-02-23

Latest versions of SSHGuard don’t use syslogd any more, it uses an internal “log sucker” that follows the logs. The default logs are “/var/log/auth.log:/var/log/maillog”, as I don’t want it to follow /var/log/mailog i override this in /etc/rc.conf with:

sshguard_watch_logs="/var/log/auth.log"

UPDATE 2014-08-05

For several reasons, I have switched from IPFW to PF. So the port to install is /usr/ports/security/sshguard-pf/ and you must add this line to your /etc/pf.conf and enable PF in /etc/rc.conf

table  persist

then to list the blocked IPs

pfctl -t sshguard -T show

to remove an IP from the list

pfctl -t sshguard -T delete aaa.bbb.ccc.ddd

to remove all the IPs

pfctl -t sshguard -T flush

NOTE

Also going to test drive on a debian box fail2ban, and will soon post quick review and differences, drawbacks, benefits versus sshguard.

SSH port forwarding

Isn’t SSH great? It’s secure and it can do lots of cool things, as providing access to services to local machines that are only available to the remote machines (that you can connect through SSH). This is called port forwarding.

Windows with Putty

So, you are on your local windows box and got ssh access to a remote machine, let’s call it “Remote” and from there you can access a service in another machine, let’s call it “Far”. The problem is that from your local windows box you can’t directly access “Far” (most times because the good people of network, and their strong sense of security…, vpn’s, etc).

So:
Localbox -> Remote (ok)
Localbox -> Far (not ok)
Localbox -> Remote -> Far (ok)

and it would be nice to test the service (lets say HTTP to exemplify) running on Far with your nice Localbox browser, instead of the console based Lynx browser that you have on Remote.

Enter the black magic of ssh port forwarding. With Putty (the SSH client for Windows) it’s pretty easy. Just open your connection normally, but before pressing the Open button, go to Connection -> SSH -> Tunnels:

The source port will be the port on your Localbox, i usually put there the localhost ip:port combination (127.0.0.1:80).  You should check with “netstat -an” if you have this free, if there is some service (IIS, Apache) already running on this ip:port stop it. The destination is the Far ip:port that you want to get access (far_ip:80). Click “Add”.  And open the connection normally and login to the Remote console. On the Localbox check again with “netstat -an” and you should have an entry like this

TCP    127.0.0.1:80           0.0.0.0:0              LISTENING

And there you go! You have an open tunnel from Localbox to Far. Now just open the browser on localbox and point it to 127.0.0.1, your request is being sent to Far. If you need an hostname to access the service correctly just put it on the hosts file:

127.0.0.1 hostname

Linux

Pretty easy… just with the ssh -L switch.
-L localport:foreig_ip:foreign_port

To make this clear, an example. On my production server i run a MySQL server instance, but it only listens to localhost (127.0.0.1) but i want to use a GUI to manage it. I have the GUI in my linux box, so it would be impossible to connect the GUI to the MySQL server… not with ssh around…

ssh user@mysqlhost -L 3306:127.0.0.1:3306

after the ssh connection is made i can access the MySQL server as if it was running on my Linux localhost. We can even check with netstat.

netstat -an | grep 3306 | grep LISTEN

it should get something like:
tcp    0    0    127.0.0.1:3306    0.0.0.0:*    LISTEN
tcp6    0    0    ::1:3306    :::*    LISTEN

There, a no-brainer sometimes very useful.