rants, tirades, ruminations

It only took a few decades, but I finally tired of looking at sshd log spam from all the break-in attempts on my various public facing devices. While fail2ban has valiantly reduced that log spam for years, the fact of the matter is, it’s still a drop in the bucket compared to the overwhelming number of source addresses from which attacks are being launched across the entirety of the public Internet. And while I’ve been using some form of two factor authentication on any of my own devices for years at this point also, the smaller the attack surface, the better, right?

I was only really interested in setting this up due to how simple nftables makes doing this. While I really like the idea of something like fwknop, I didn’t want yet another privileged service (especially one running a perpetual packet capture on my interfaces essentially) on any of my devices. It’s worth noting here that all of the following will potentially break horrifically if you keep the default fail2ban SSH jail enabled as the nc command used in my knock script to test whether SSH is currently open will show up as premature disconnects during the preauth stage, and fail2ban will ban your IP after a few of these. This is only really an issue if you’re running back to back commands in fairly short succession. But I was running into this exact problem with some of my backup scripts, so you’ll either need to modify your fail2ban filters or disable the SSH jail entirely to avoid this biting you in the ass. Having said that, since your log traffic for anything SSH will pretty much fall off a cliff after implementing this, it probably doesn’t matter too much if you disable fail2ban entirely unless you’re also using it for other services.

Let’s look at a basic nftables configuration which implements some sane defaults and also includes our port knocking logic:

flush ruleset

define guarded_ports = {ssh}

table inet filter {
	set clients_ipv4 {
		type ipv4_addr
		flags timeout
	}

	set clients_ipv6 {
		type ipv6_addr
		flags timeout
	}

	set candidates_ipv4 {
		type ipv4_addr . inet_service
		flags timeout
	}

	set candidates_ipv6 {
		type ipv6_addr . inet_service
		flags timeout
	}

	chain input {
		type filter hook input priority 0; policy reject;

		# refresh port knock timer for existing SSH connections
		#tcp dport ssh ct state established ip  saddr @clients_ipv4 update @clients_ipv4 { ip  saddr timeout 10s }
		#tcp dport ssh ct state established ip6 saddr @clients_ipv6 update @clients_ipv6 { ip6 saddr timeout 10s }

		# established/related connections
		ct state established,related accept

		# invalid connections
		ct state invalid reject

		# loopback interface
		iif lo accept

		# ICMPv6 packets which must not be dropped, see https://tools.ietf.org/html/rfc4890#section-4.4.1
		meta nfproto ipv6 icmpv6 type { destination-unreachable, echo-reply, echo-request, nd-neighbor-advert, nd-neighbor-solicit, nd-router-advert, nd-router-solicit, packet-too-big, parameter-problem, time-exceeded, 148, 149 } accept # Certification Path Solicitation (148) / Advertisement (149) Message RFC3971
		ip6 saddr fe80::/10 icmpv6 type { 130, 131, 132, 143, 151, 152, 153 } accept
		# Multicast Listener Query (130) / Report (131) / Done (132) RFC2710
		# Version 2 Multicast Listener Report (143) RFC3810
		# Multicast Router Advertisement (151) / Solicitation (152) / Termination (153)
		ip protocol icmp icmp type { destination-unreachable, echo-reply, echo-request, parameter-problem, router-advertisement, source-quench, time-exceeded } accept

		# port knocking for SSH
		# accept any local LAN SSH connections
		ip saddr 192.168.1.0/24 tcp dport ssh accept # 22
		tcp dport 12345 add @candidates_ipv4 {ip  saddr . 23456 timeout 2s}
		tcp dport 12345 add @candidates_ipv6 {ip6 saddr . 23456 timeout 2s}
		tcp dport 23456 ip  saddr . tcp dport @candidates_ipv4 add @candidates_ipv4 {ip  saddr . 34567 timeout 2s}
		tcp dport 23456 ip6 saddr . tcp dport @candidates_ipv6 add @candidates_ipv6 {ip6 saddr . 34567 timeout 2s}
		tcp dport 34567 ip  saddr . tcp dport @candidates_ipv4 add @candidates_ipv4 {ip  saddr . 45678 timeout 2s}
		tcp dport 34567 ip6 saddr . tcp dport @candidates_ipv6 add @candidates_ipv6 {ip6 saddr . 45678 timeout 2s}
		tcp dport 45678 ip  saddr . tcp dport @candidates_ipv4 update @clients_ipv4 {ip  saddr timeout 10s} log prefix "Successful portknock: "
		tcp dport 45678 ip6 saddr . tcp dport @candidates_ipv6 update @clients_ipv6 {ip6 saddr timeout 10s} log prefix "Successful portknock: "

		tcp dport $guarded_ports ip  saddr @clients_ipv4 counter accept
		tcp dport $guarded_ports ip6 saddr @clients_ipv6 counter accept
		tcp dport $guarded_ports ct state established,related counter accept

		tcp dport $guarded_ports counter reject with tcp reset

		# reject everything else but be friendly!
		counter reject with icmp type host-unreachable
	}

	chain output {
		type filter hook output priority 100; policy accept;
	}

	chain forward {
		type filter hook forward priority 0; policy reject;
	}
}

This is pretty much your standard firewall configuration to block everything inbound or forwarded and allow everything outbound, with a few extra bits and pieces to hopefully remain a friendly network neighbor and keep things like ICMP and IPv6 working correctly. Most of the port knocking stuff is identical to the example on the nftables’ wiki, with a couple of notable changes.

One problem I had with this setup was that some of my backup scripts which were making back to back connections to hosts were sometimes failing. This was odd because as we’ll see below, all of my SSH connections were being proceeded as you’d expect with a knock command which should have meant that I had a 10 second window to ultimately connect to the host before the SSH port was again closed. The reason that wasn’t happening was because if my backups happened to take less than 10 seconds, then a knock command proceeding the following SSH command wasn’t actually extending the timeout for my source IP address in the relevant clients’ set. So my knock script would rightfully report that SSH was open and then my actual SSH connection would end up failing moments later when it tried to initiate the connection and find the port closed.

The first solution I implemented are the two commented out lines about refreshing the port knock timer, both using the update directive. I’ve left those here in case someone wants this version of the functionality. The way those are written, it will keep updating the timeout as long as any existing SSH connections are open. However, this also means that anyone else coming from the same source IP can attempt to connect for as long as you have your own SSH connections open. This wasn’t exactly what I wanted as I actually liked the fact that SSH was only open for 10 seconds after a successful knock attempt, and then it was closed back down entirely except for any already established connections.

The solution therefore was to simply change the ‘add @clients_ipv[46]’ statements to use update instead. This means that any successful port knock will give you a full 10 seconds instead of whatever small amount of time may have remained from the previous successful port knock, thereby hopefully avoiding this sort of race condition effect I was seeing during my backups.

The remaining pieces to make this work are the knock script itself and altering your SSH configuration to use the knock script correctly. Thankfully, someone else had already figured out the cleanest way to configure SSH to work in this sort of setup. While I’m not a massive fan of needing to run all of my connections through nc for any hosts where I’ve implemented this, I’ve been using similar SSH configurations for years at this point (prior to the ProxyJump directive anyway), so it doesn’t really bother me:

Host tango
  Hostname tango.example.com
  ProxyCommand bash -c '/home/user/bin/knock %h %p 12345 23456 34567 45678; exec nc %h %p'

That is written generically enough that if you use a non-standard SSH port, it should still work. And since my knock script is already checking to ensure the service port is open, I’m skipping the sleep statement used in the source linked above.

And finally, here’s the zsh knock script I’ve concocted:

#!/usr/bin/env zsh

# load module to parse command line arguments
zmodload zsh/zutil
zparseopts -D -E -A opts -- h x

# load module to avoid use of GNU sleep
zmodload zsh/zselect

# enable XTRACE shell option for full debugging output of scripts
if (( ${+opts[-x]} )); then
	set -x
fi

# display short command usage
if [[ -z "${2}" ]] || (( ${+opts[-h]} )); then
	echo "usage: ${0:t} [ -h ] [ -x ] host port [ knock_port ] .." >&2
	echo -e '\n\t-h\tshow this help\n\t-x\tenable shell debugging' >&2
	echo -e '\thost\tdestination host name' >&2
	echo -e '\tport\tdestination service port\n' >&2
	echo -e 'Specifying no knock_port(s) will use 12345 23456 34567 45678 by default.\n' >&2
	exit 1
fi

# define our variables
host="${1}"
port="${2}"
shift 2
knock_ports="${@:-12345 23456 34567 45678}"
attempts=1

# helper function to check whether the service port is actually open
function check_service_port {
	if nc -w1 ${host} ${port} &> /dev/null <& -; then
		exit 0
	fi
}

# check if the service port is already open for some reason
# commented out to avoid race condition and need for additional firewall update rule
#check_service_port

# main loop to open requested service port via port knocking
while [[ ${attempts} -lt 9 ]]; do

	for knock_port in ${=knock_ports}; do
		nc -w1 ${host} ${knock_port} &> /dev/null <& - &
		# increasingly back off on subsequent attempts in case packets are arriving out of order due to high latency
		zselect -t ${attempts}0
	done

	# check now if the service port is open
	check_service_port
	# if not, try again
	((attempts+=1))

done

# all attempts failed, so exit with error
exit 1

And that should be all there is to it! Obviously, this can be used potentially for any service and not just SSH. And depending on traffic conditions between the source and destination, you might need to adjust the number of attempts or the back off timer logic, where zselect is in hundreths of a second, hence adding the zero to the attempt number in the script logic. Similarly, you could increase or decrease the number of required ports in your knock sequence by adjusting the nftables configuration appropriately and passing the requisite number of knock_port arguments to the knock script.

It’s also worth noting that anyone sniffing traffic between your source and destination would potentially be able to discern your port knock sequence which is certainly one of the advantages of something like fwknop. However, again, wanting to avoid the added complexity of running an additional service, this solution is good enough for me, combined with all of my other security mechanisms already in place. Besides, if I started seeing a lot of successful port knocking messages in my logs from IP’s other than my own, I’d also now be aware that something really bad has happened somewhere between my source and destination hosts which would require immediate investigation.

Anyway, hopefully this proves useful to someone else. I’ve certainly enjoyed the total abatement of SSH related log spam as a result of implementing this!