blog of the mundane...

NixOS laptop homelab

Wed, 13 Nov 2024 02:06:19 +0000

The Why

For years I’ve put off building any kind of homelab setup. Mostly due to my unwillingness to “compromise” on using non-x86 based devices and therefore ultimately, cost.

I had been keeping an eye out for a long time since the actual price of just any x86 based system has been fairly affordable for a long time. A lot of that affordability though, came in the form of older Haswell to Skylake era Intel mini PC’s or one of the newer, but much lower end Atom or other similarly low powered, lower performing processors. I almost pulled the trigger on some well priced Lenovo ThinkCentre mini PC’s, and I might still end up there at some point.

As I was browsing though, I threw laptops into the mix and happened across an Ebay listing with four remaining Lenovo ThinkPad L14 Gen 1 Ryzen based systems (model 20U6S1VW00 specifically) for only $208 each when buying all four. Since they each came with 32GB of DDR4 RAM and a 512GB NVMe drive right out of the gate, as well as a more than competent 6-core/12-thread AMD Ryzen 5 Pro 4650U, I grabbed all four! They’ve got three PCIe 3.0 x4 slots (2x M.2 2280 and 1x M.2 2242) and support up to 64GB of RAM. As a starting point, they seem like a risk free investment which I can always hand out to folks (or myself!) as replacement laptops if I end up not finding much use for them.

I’ve spent the last few days piecing together all the things which I think will make this an excellent homelab setup for me. And a fair amount of that is being facilitated by NixOS, as we’ll see.

So what did I want to be able to do exactly? The goal was to have gear that functioned essentially the same as enterprise gear that I could easily provision however I wanted anytime. I knew I would need to leverage my existing DHCP server on my NixOS router to pass along PXE information of some kind to netboot into a live Linux environment from which I would then be able to provision each machine however I chose. I also wanted to be able to send Wake-on-LAN requests to wake up the machines so they wouldn’t need to run 24x7, but turning them back on would be a cinch.

The What

A brief note about NixOS before I dive into the technical stuff. This isn’t the place to learn all things Nix. However, as I’ve experienced over the past year on my own Nix journey, the more examples of various styles of Nix incantations out in the wild, the more likely you’ll see someone doing something that finally makes things click for you about how all of this seeming magic actually works.

And NixOS lets you do some really incredible stuff. I’m not sure if it is, in fact, the one and only way to put together any operating system. But the more I fall into it, the more I feel like my mind is opening up to some greater knowledge of how systems are meant to be designed. The modular design and the level of functionality in the tooling are standout, best in class features.

All of which is to say, you could certainly do all of what follows with any number of other pieces of software. But the idea of using Nix to build an enterprise cluster of some kind seems like it should end up being far easier to capture and maintain long term as a NixOS configuration. Then we get all the infrastructure as code benefits that NixOS brings without slogging through the endless morass of Helm charts I see in my professional life. Maybe we’ll still end up there in the end? But even if we do, I want at least an incredibly easily reproducible base from which to start whenever I want and all mostly at the push of a remote button.

Here are just a few of the places I referenced as I worked my way through this:

I’m not sure that NixOS has a documentation problem as much as it does a flexibility problem. There’s just a lot of different ways to arrive at the same configuration ultimately. But as I said earlier, the more people document their various recipes, the easier I think it will be in the future for people to continue adopting and contributing to Nix.

Having said that, I’m going to start dropping a bunch of Nix configuration snippets at this point. Most of them will be just that, snippets. You’ll need to fit them into however you’re doing things within your own configuration. I am using flakes here, so all those usual qualifiers apply. Feel free to come back later and pick through whichever pieces you might find useful if you’re not yet at a point where any of this makes sense yet. I’m not going to pretend like any kind of subject matter expert here. It’s a lot to wrap your head around. But here’s a link to my paltry efforts, if you should like to try to piece together everything below with all the rest:

https://arrakis.bitgnome.net/nipsy/git/nix/

I’m only using a handful of sops-nix secrets, so most everything is in the clear.

The How

DHCP, PXE and iPXE

My NixOS router, darkstar, was already running kea for DHCP services. I saw folks using Pixiecore elsewhere to supplement all the other pieces necessary at this point. But I wanted to follow a more familiar design and provide the PXE information myself directly from my DHCP server:

  services.kea.dhcp4 = {
    enable = true;
    settings = {
      interfaces-config.interfaces = [ "enp116s0" ];

      lease-database = {
        name = "/var/lib/kea/dhcp4.leases";
        persist = true;
        type = "memfile";
      };

      renew-timer = 900;
      rebind-timer = 1800;
      valid-lifetime = 3600;

This starts off with a fairly normal Nix service enablement block which quickly morphs into virtually the exact JSON style configuration file syntax kea expects with equal signs instead of colons.

We start by binding the service to the router’s internal LAN interface, telling kea where and how to store its lease state, and some basic DHCP parameters that will determine how often clients need to request a new DHCP lease.

We continue:

      option-data = [
        {
          name = "domain-name-servers";
          data = "192.168.1.1";
          always-send = true;
        }

        {
          name = "domain-name";
          data = "bitgnome.net";
          always-send = true;
        }

        {
          name = "ntp-servers";
          data = "192.168.1.1";
          always-send = true;
        }
      ];

Another fairly standard block. I’ve got a single, flat network right now so these are the options I’m handing out in every DHCP offer.

      client-classes = [
        {
          name = "XClient_iPXE";
          test = "substring(option[77].hex,0,4) == 'iPXE'";
          boot-file-name = "http://arrakis.bitgnome.net/boot/netboot.ipxe";
        }

        {
          name = "UEFI-64-1";
          test = "substring(option[60].hex,0,20) == 'PXEClient:Arch:00007'";
          next-server = "192.168.1.1";
          boot-file-name = "/etc/tftp/ipxe.efi";
        }

        {
          name = "UEFI-64-2";
          test = "substring(option[60].hex,0,20) == 'PXEClient:Arch:00008'";
          next-server = "192.168.1.1";
          boot-file-name = "/etc/tftp/ipxe.efi";
        }

        {
          name = "UEFI-64-3";
          test = "substring(option[60].hex,0,20) == 'PXEClient:Arch:00009'";
          next-server = "192.168.1.1";
          boot-file-name = "/etc/tftp/ipxe.efi";
        }

        {
          name = "Legacy";
          test = "substring(option[60].hex,0,20) == 'PXEClient:Arch:00000'";
          next-server = "192.168.1.1";
          boot-file-name = "/etc/tftp/undionly.kpxe";
        }
      ];

Here’s the kea side of the PXE configuration on the local network segment (remember, in Nix language format, not actually kea JSON format!). If any client making a DHCP request matches one of these test cases, then the extra options provided are passed along as part of the DHCP offer. This covers not only the initial PXE boot by any matching client but also the subsequent iPXE boot we chain into from the PXE boot environment. iPXE will reference the URL provided in the first block for which files to download now via HTTP instead of the much slower PXE for how to continue booting into the custom NixOS installer image later.

      subnet4 = [
        {
          id = 1;
          subnet = "192.168.1.0/24";
          pools = [ { pool = "192.168.1.100 - 192.168.1.199"; } ];

          option-data = [
            {
              name = "routers";
              data = "192.168.1.1";
            }
          ];

          reservations = [
            ({ hw-address = "8c:8c:aa:4e:e9:8c"; ip-address = "192.168.1.11"; }) # jupiter
            ({ hw-address = "38:f3:ab:59:06:e0"; ip-address = "192.168.1.12"; }) # saturn
            ({ hw-address = "8c:8c:aa:4e:fc:aa"; ip-address = "192.168.1.13"; }) # uranus
            ({ hw-address = "38:f3:ab:59:08:10"; ip-address = "192.168.1.14"; }) # neptune
          ];
        }
      ];
    };
  };

Lastly, again, a fairly standard block to define the subnet range to allocate to DHCP clients as well as static reservations, including the Sun planet themed bunch of newly acquired Lenovo laptops, jupiter through neptune.

Along with the above, you’ll probably want the following block nearby to handle the rest of the PXE heavy lifting as well as the initial iPXE work:

  environment = {
    etc = {
      "tftp/ipxe.efi".source = "${pkgs.ipxe}/ipxe.efi";
      "tftp/undionly.kpxe".source = "${pkgs.ipxe}/undionly.kpxe";
    };

    networking.firewall.interfaces.enp116s0.allowedUDPPorts = [ 69 ];

    systemPackages = with pkgs; [
      ipxe
      tftp-hpa
      wol
    ];
  };

  systemd.services = {
    tftpd = {
      after = [ "nftables.service" ];
      description = "TFTP server";
      serviceConfig = {
        User = "root";
        Group = "root";
        Restart = "always";
        RestartSec = 5;
        Type = "exec";
        ExecStart = "${pkgs.tftp-hpa}/bin/in.tftpd -l -a 192.168.1.1:69 -P /run/tftpd.pid /etc/tftp";
        TimeoutStopSec = 20;
        PIDFile = "/run/tftpd.pid";
      };
      wantedBy = [ "multi-user.target" ];
    };
  };

This is adding a few useful packages along with creating a tftpd service using tftp-hpa’s in.tftpd. It’s also building a tftp path from which to serve requested files in /etc/tftp. In retrospect, I can probably point directly into the /nix/store in that ExecStart line similar to what I do later with the netboot image. I chose to use tftp-hpa over the stock inbuilt netkittftp (if using the services.tftpd.enable option) as I’m more familiar and I preferred running the service directly rather than use xinetd which is how netkittftp is configured to work via that services.tftpd option.

Now, I wasted a significant portion of a day around this point doing what I thought was going to be a really straightforward PXE boot setup. It turns out, the machine I’ve named jupiter here happens to have a busted PXE client built into its Realtek NIC. I thought I was losing my mind because it worked once, which as I was to discover, is about all you ever get out of it, and then it seemingly rarely works again. After much frustration and some really manic deep diving into things like bpftrace to watch for file access and process spawning, I finally broke down and pulled out a different laptop to try which, as it so happened, worked flawlessly every time just like the other two. So, if anyone has any suggestions as to why this might happen or how to fix it long term, I’d love to hear about it. I did notice the Realtek PXE driver client in the Lenovo BIOS mentions that it is beta! But I don’t see any available firmware updates for that through LVFS at least. Maybe I can update them under a running version of Windows (already licensed for it conveniently) with some Realtek executable?

Anyway, the short term workaround I subsequently discovered after finally realizing the actual problem was to use fwupdmgr to reinstall the latest Lenovo BIOS on jupiter. After the initial application of the BIOS, the next PXE boot has always worked thus far. It may not work more than once and often breaks within the first few attempts. But at least I know how to mitigate the issue in a way that doesn’t seem to massively interrupt any of the rest of this workflow.

I’ve also included my firewall rule here for the tftpd server. I don’t think I’m specifically opening the DHCP ports themselves anywhere as I’m also using the inbuilt nftables ruleset via networking.nftables.enable = true; which seems to cover that.

So now we’re responding to DHCP requests and providing PXE clients with the data pointing them to download the iPXE image. Once the inbuilt PXE client boots into the iPXE client provided, it then also performs a DHCP operation where it is now given a URL to load.

iPXE and nginx

The URL in question needs to be served from some HTTP server. I’m already running nginx elsewhere on my internal network, so that’s where I’m hosting both the iPXE script that is loaded by each iPXE client and the netboot data itself to actually boot into a remotely accessible NixOS installer environment. I won’t provide my entire nginx configuration here enabling all of the SSL stuff via Let’s Encrypt, but you can refer to my repo to find all of that in this same file probably still:

  services.nginx = let

      sys = lib.nixosSystem {
        system = "x86_64-linux";

        modules = [
          ({ config, pkgs, lib, modulesPath, ... }: {
            imports = [
              (modulesPath + "/installer/netboot/netboot-minimal.nix")
              ../common/optional/services/nolid.nix
            ];

            config = {
              environment.systemPackages = with pkgs; [
                git
                rsync
              ];

              nix.settings.experimental-features = [ "nix-command" "flakes" ];

              services.openssh = {
                enable = true;
                openFirewall = true;

                settings = {
                  PasswordAuthentication = false;
                  KbdInteractiveAuthentication = false;
                };
              };

              users.users = {
                nixos.openssh.authorizedKeys.keys = [ (builtins.readFile ../common/users/nipsy/keys/id_arrakis.pub) ];
                root.openssh.authorizedKeys.keys = [ (builtins.readFile ../common/users/nipsy/keys/id_arrakis.pub) ];
              };
            };
          })
        ];
      };

      build = sys.config.system.build;

    in {

Wait, what? Okay, so I started down this path initially by figuring out how to create a custom NixOS ISO image. And you can still find that logic in my repository along with a handy zsh alias (geniso) I created so I wouldn’t have to type the entire command.

However, why bother with that crap when I can just inject the custom built netboot artifacts directly into my nginx configuration itself?

That’s what is happening here. You’ll see all the usual config options you’d see to configure a normal, running system, along with injecting my own personal SSH keys for both the root and nixos users in the resulting netboot image and installing some handy additional commands which could prove useful in the installer environment. But in this instance, we are doing all that work dynamically under a variable named build which then has its resulting built artifacts referenced here where the service.nginx block actually begins:

      appendHttpConfig = ''
        geo $geo {
                default 0;
                127.0.0.1 1;
                ::1 1;
                192.168.1.0/24 1;
        }

        map $scheme $req_ssl {
                default 1;
                http 0 ;
        }

        map "$geo$req_ssl" $force_enable_ssl {
                default 0;
                00 1;
        }
      '';
      enable = true;

      recommendedGzipSettings = true;
      recommendedOptimisation = true;
      #recommendedProxySettings = true;
      recommendedTlsSettings = true;

      sslCiphers = "AES256+EECDH:AES256+EDH:!aNULL";

      virtualHosts = {
        "arrakis.bitgnome.net" = {
          addSSL = true;
          enableACME = true;

          extraConfig = ''
            if ($force_enable_ssl) {
                return 301 https://$host$request_uri;
            }
          '';

          locations = {
            "= /boot/bzImage" = {
              alias = "${build.kernel}/bzImage";
            };

            "= /boot/initrd" = {
              alias = "${build.netbootRamdisk}/initrd";
            };

            "= /boot/netboot.ipxe" = {
              alias = "${build.netbootIpxeScript}/netboot.ipxe";
            };

            "/" = {
              tryFiles = "$uri $uri/ =404";
            };
          };

          root = "/var/www";
        };
      };
    };

As mentioned above, this is where the references to the netboot artifacts get filled in with aliases pointing directly into /nix/store. And the especially great thing about all of this, is that the netboot image is kept perpetually up to date and these references should always be pointing at the latest version. The rest will be cleaned up automatically by your next scheduled garbage collection once they’re no longer referenced. I discovered the syntax for these exact location definitions (using the “= /…” style for each location name) by cheating and looking at how the cgit module accomplished the same thing. It wasn’t until sometime after looking at that module’s code I finally understood how the equal sign is being used to build these names.

And you don’t even need to learn how to write iPXE scripts, because again, the netboot build process generates one for us which we can then drop in directly as a reference in our nginx configuration. And anytime anything changes, it all gets updated automatically and nginx reloaded accordingly! Neat stuff.

We finally have our custom NixOS installer up and running hopefully and we should be able to log in directly as root.

Install

I wrote a shell script for the next part, which I’ve also dropped in my repo at the top under scripts/remote-install-with-disko. Here’s the basic command sequence from that script though:

# 192.168.1.11 is jupiter per the above static reservation
ssh root@192.168.1.11 nix run github:nix-community/disko/latest -- --mode disko --flake https://arrakis.bitgnome.net/nipsy/git/nix/snapshot/nix-master.tar#jupiter
ssh root@192.168.1.11 nixos-install --flake https://arrakis.bitgnome.net/nipsy/git/nix/snapshot/nix-master.tar#jupiter
ssh root@192.168.1.11 reboot

In reality, I’m also using split-horizon DNS with unbound on darkstar to provide full DNS resolution for these LAN based devices. You can find that all in the repo also. But we don’t really need that here.

Two commands. That’s it. The first leverages the wonderful community created disko project to handle the formatting and mounting of all the drives as defined under hosts/jupiter/disks.nix. This configuration then also gets consumed and referenced during the subsequent nixos-install command to define all the file system mounts in /etc/fstab on the running system.

You of course need to define the system configuration for jupiter and all the rest in your flake.

Jupiter and Beyond the Infinite

It’s worth talking a little about the laptop NixOS configurations themselves. I’m not going to drop the entire configuration here for jupiter or any of the rest. You can go look at them easily enough in the repo, and your layout might be sufficiently different from mine that you can’t just drop mine in easily.

But some of the more important pieces include the lid handling since these are laptops and the Wake-on-LAN functionality. The lid handling was easy, and the netboot image also includes this bit as to avoid any nasty surprises by virtue of the fact all four laptops are stacked on top of one another with their lids closed:

  services.logind = {
    lidSwitch = "ignore";
    lidSwitchDocked = "ignore";
    lidSwitchExternalPower = "ignore";
  };

The Wake-on-LAN was even simpler:

  networking.interfaces.enp2s0f0.wakeOnLan.enable = true;

Now, this does have a corresponding option in the BIOS which I have enabled when the AC adapter is connected. I also enabled the BIOS option to power on automatically whenever AC power is restored.

And since the BIOS came up, it’s also worth mentioning the boot order. I decided to only keep two boot entries active, the first NVMe drive followed by the Realtek IPv4 PXE client. There’s also an option to define which boot entry to use when booting via Wake-on-LAN, and I also set that to the first NVMe drive. The thinking here being, all I need to do to wipe and reinstall a machine (by forcing it to PXE boot next time), is:

umount /boot && mkfs.vfat /dev/nvme0n1p1 && reboot

which wipes my EFI boot partition and reboots, forcing the PXE client to boot when the first NVMe option fails to boot correctly. I’ve tested this and it works brilliantly.

And while we’re talking about WoL, you might have already noticed the wol package installed on darkstar earlier. Once the laptops are configured for it in BIOS and you have a working OS on them to set the NIC into the correct mode (as done above), you can run this from the router (or whatever other LAN attached device you want):

wol -vi 192.168.1.255 8c:8c:aa:4e:e9:8c

to wake up jupiter for instance from a power off state.

Where?

Where to next? I’m definitely going to configure VXLAN on top of these once I get some patch cords for all four laptops to connect them up to the switch sitting right next to them. I’d also like to see how terrible something like OpenStack might be to get up and running in a declarative manner. I’ll probably end up throwing in some extra storage somewhere along the way so I can play with Ceph a bit too. If these machines end up being too limiting, it looks like the Lenovo ThinkCentre M75q’s are available in this same general price range and specification and also include a 2.5” bay for even larger storage options.

wireless bridge with proxy ARP, nftables, WireGuard, a network namespace and veth pair

Sun, 12 Mar 2023 12:56:43 +0000

I recently had to move my home NAS device onto a wireless connection. In doing so, I wanted to maintain my use of a network namespaces and veth pair to isolate all of my WireGuard VPN traffic. However, under Linux, setting up what is effectively a bridged connection on a wireless interface has never been as easy as some other operating systems. As I’m not sure when I might finally be able to get a decent wired connection hooked back up to my NAS, it came time to figure out how to make all of this work together.

The starting point was the Bridging Network Connections with Proxy ARP page on the Debian wiki. It had the very basics of getting this set up, but was missing most of the details. After messing with things a bit, I finally ended up with a working configuration. Let’s start with the very basics covered on the wiki. Since I’m old and crotchety, I’m still using /etc/sysctl.conf directly:

net.ipv4.ip_forward=1
net.ipv4.conf.all.proxy_arp=1

My motherboard happened to come with an Intel AX200 wireless interface. I’m currently using wpa_supplicant via systemd to statically configure my device for my home wireless network. I also have iwd currently installed although disabled, so my wireless interface’s normal name is being overridden to wlan0 thanks to the iwd package dropping /lib/systemd/network/80-iwd.link in place. When configuring wpa_supplicant@wlan0.service in systemd, it’s looking for the corresponding configuration file at /etc/wpa_supplicant/wpa_supplicant-wlan0.conf:

network={
	ssid='Super Secret SSID'
	bssid=12:34:56:78:9a:bc
	key_mgmt=SAE
	sae_password="super secret password"
	ieee80211w=2
}

I had to enable that service obviously with systemctl enable wpa_supplicant@wlan0.service since it isn’t configured by default.

Next up was to configure the interface itself. As previously mentioned, since I’m an old man, I’m also still using /etc/network/interfaces, which contains the following relevant section now:

allow-hotplug wlan0
iface wlan0 inet static
	address 192.168.99.2
	gateway 192.168.99.1
	netmask 255.255.255.0
	post-up ip netns add vpn
	post-up ip link add veth.host type veth peer veth.vpn
	post-up ip link set dev veth.host up
	post-up ip link set veth.vpn netns vpn up
	post-up ip -n vpn address add 192.168.99.3/24 dev veth.vpn
	post-up ip route add 192.168.99.3/32 dev veth.host
	post-up ip link add wg1 type wireguard
	post-up ip link set wg1 netns vpn
	post-up ip -n vpn -4 address add 172.24.0.10/32 dev wg1
	post-up ip netns exec vpn wg setconf wg1 /etc/wireguard/wg1.conf
	post-up ip -n vpn link set wg1 up
	post-up ip -n vpn route add default dev wg1
	post-up ip netns exec vpn nft -f /etc/nftables-vpn.conf

The DHCP range starts higher up in my LAN, so I’m using the first couple of addresses at the bottom of the network after my gateway device for my NAS and this separate, VPN only network namespace. The WireGuard interface is configured within the namespace using the appropriate static address as supplied by my VPN provider. I have a separate WireGuard wg0 already configured outside the namespace, so wg1 is what we’re using inside of it.

And the final piece listed their at the end of the interface setup loads in my firewall rules for the namespace from /etc/nftables-vpn.conf:

# VPN firewall

flush ruleset

table inet filter {
	chain input {
		type filter hook input priority filter; policy drop;

		# established/related connections
		ct state established,related accept

		# invalid connections
		ct state invalid drop

		# loopback interface
		iif lo accept

		# ICMP (routers may also want: mld-listener-query, nd-router-solicit)
		#ip6 nexthdr icmpv6 icmpv6 type { destination-unreachable, echo-reply, echo-request, nd-neighbor-advert, nd-neighbor-solicit, nd-router-advert, packet-too-big, parameter-problem, time-exceeded } accept
		ip protocol icmp icmp type { destination-unreachable, echo-reply, echo-request, parameter-problem, router-advertisement, source-quench, time-exceeded } accept

		# services
		iif veth.vpn tcp dport 9091 accept # Transmission
		iif veth.vpn tcp dport 9117 accept # Jackett
		iifname wg1 tcp dport { 49152-65535 } accept # Transmission
	}

	chain output {
		type filter hook output priority filter; policy drop;

		# explicitly allow my DNS traffic without VPN
		skuid nipsy ip daddr 192.168.99.1 tcp dport domain accept
		skuid nipsy ip daddr 192.168.99.1 udp dport domain accept

		# explicitly allow my Transmission or Jackett RPC traffic without VPN
		oifname veth.vpn skuid nipsy tcp sport 9091 accept
		oifname veth.vpn skuid nipsy tcp sport 9117 accept

		# allow any traffic out through VPN
		oifname wg1 accept

		# drop everything else
		counter drop
	}

	chain forward {
		type filter hook forward priority filter; policy drop;
	}
}

Both Jackett and Transmssion are configured via pretty basic systemd service files you might find anywhere else, with the notable exception that both include the following in their [Service] section:

NetworkNamespacePath=/run/netns/vpn

which seems to do the trick for running them in the namespace correctly.

Lastly, if you create the namespace before setting the kernel parameters above via sysctl, the namespace will not inherit those settings. You’ll probably need to either restart or delete and recreate the namespace for those values to be inherited properly. I’m honestly not sure if they’re specifically relevant within the context of the namespace itself since I think they only apply to what’s happening at the host level directly on wlan0, but it’s worth mentioning if everything else looks right and it’s still not working for you. Best of luck!

nftables and port knocking

Fri, 18 Mar 2022 19:51:59 +0000

It only took a few decades, but I finally tired of looking at sshd log spam from all the break-in attempts on my various public facing devices. While fail2ban has valiantly reduced that log spam for years, the fact of the matter is, it’s still a drop in the bucket compared to the overwhelming number of source addresses from which attacks are being launched across the entirety of the public Internet. And while I’ve been using some form of two factor authentication on any of my own devices for years at this point also, the smaller the attack surface, the better, right?

I was only really interested in setting this up due to how simple nftables makes doing this. While I really like the idea of something like fwknop, I didn’t want yet another privileged service (especially one running a perpetual packet capture on my interfaces essentially) on any of my devices. It’s worth noting here that all of the following will potentially break horrifically if you keep the default fail2ban SSH jail enabled as the nc command used in my knock script to test whether SSH is currently open will show up as premature disconnects during the preauth stage, and fail2ban will ban your IP after a few of these. This is only really an issue if you’re running back to back commands in fairly short succession. But I was running into this exact problem with some of my backup scripts, so you’ll either need to modify your fail2ban filters or disable the SSH jail entirely to avoid this biting you in the ass. Having said that, since your log traffic for anything SSH will pretty much fall off a cliff after implementing this, it probably doesn’t matter too much if you disable fail2ban entirely unless you’re also using it for other services.

Let’s look at a basic nftables configuration which implements some sane defaults and also includes our port knocking logic:

flush ruleset

define guarded_ports = {ssh}

table inet filter {
	set clients_ipv4 {
		type ipv4_addr
		flags timeout
	}

	set clients_ipv6 {
		type ipv6_addr
		flags timeout
	}

	set candidates_ipv4 {
		type ipv4_addr . inet_service
		flags timeout
	}

	set candidates_ipv6 {
		type ipv6_addr . inet_service
		flags timeout
	}

	chain input {
		type filter hook input priority 0; policy reject;

		# refresh port knock timer for existing SSH connections
		#tcp dport ssh ct state established ip  saddr @clients_ipv4 update @clients_ipv4 { ip  saddr timeout 10s }
		#tcp dport ssh ct state established ip6 saddr @clients_ipv6 update @clients_ipv6 { ip6 saddr timeout 10s }

		# established/related connections
		ct state established,related accept

		# invalid connections
		ct state invalid reject

		# loopback interface
		iif lo accept

		# ICMPv6 packets which must not be dropped, see https://tools.ietf.org/html/rfc4890#section-4.4.1
		meta nfproto ipv6 icmpv6 type { destination-unreachable, echo-reply, echo-request, nd-neighbor-advert, nd-neighbor-solicit, nd-router-advert, nd-router-solicit, packet-too-big, parameter-problem, time-exceeded, 148, 149 } accept # Certification Path Solicitation (148) / Advertisement (149) Message RFC3971
		ip6 saddr fe80::/10 icmpv6 type { 130, 131, 132, 143, 151, 152, 153 } accept
		# Multicast Listener Query (130) / Report (131) / Done (132) RFC2710
		# Version 2 Multicast Listener Report (143) RFC3810
		# Multicast Router Advertisement (151) / Solicitation (152) / Termination (153)
		ip protocol icmp icmp type { destination-unreachable, echo-reply, echo-request, parameter-problem, router-advertisement, source-quench, time-exceeded } accept

		# port knocking for SSH
		# accept any local LAN SSH connections
		ip saddr 192.168.1.0/24 tcp dport ssh accept # 22
		tcp dport 12345 add @candidates_ipv4 {ip  saddr . 23456 timeout 2s}
		tcp dport 12345 add @candidates_ipv6 {ip6 saddr . 23456 timeout 2s}
		tcp dport 23456 ip  saddr . tcp dport @candidates_ipv4 add @candidates_ipv4 {ip  saddr . 34567 timeout 2s}
		tcp dport 23456 ip6 saddr . tcp dport @candidates_ipv6 add @candidates_ipv6 {ip6 saddr . 34567 timeout 2s}
		tcp dport 34567 ip  saddr . tcp dport @candidates_ipv4 add @candidates_ipv4 {ip  saddr . 45678 timeout 2s}
		tcp dport 34567 ip6 saddr . tcp dport @candidates_ipv6 add @candidates_ipv6 {ip6 saddr . 45678 timeout 2s}
		tcp dport 45678 ip  saddr . tcp dport @candidates_ipv4 update @clients_ipv4 {ip  saddr timeout 10s} log prefix "Successful portknock: "
		tcp dport 45678 ip6 saddr . tcp dport @candidates_ipv6 update @clients_ipv6 {ip6 saddr timeout 10s} log prefix "Successful portknock: "

		tcp dport $guarded_ports ip  saddr @clients_ipv4 counter accept
		tcp dport $guarded_ports ip6 saddr @clients_ipv6 counter accept
		tcp dport $guarded_ports ct state established,related counter accept

		tcp dport $guarded_ports counter reject with tcp reset

		# reject everything else but be friendly!
		counter reject with icmp type host-unreachable
	}

	chain output {
		type filter hook output priority 100; policy accept;
	}

	chain forward {
		type filter hook forward priority 0; policy reject;
	}
}

This is pretty much your standard firewall configuration to block everything inbound or forwarded and allow everything outbound, with a few extra bits and pieces to hopefully remain a friendly network neighbor and keep things like ICMP and IPv6 working correctly. Most of the port knocking stuff is identical to the example on the nftables’ wiki, with a couple of notable changes.

One problem I had with this setup was that some of my backup scripts which were making back to back connections to hosts were sometimes failing. This was odd because as we’ll see below, all of my SSH connections were being proceeded as you’d expect with a knock command which should have meant that I had a 10 second window to ultimately connect to the host before the SSH port was again closed. The reason that wasn’t happening was because if my backups happened to take less than 10 seconds, then a knock command proceeding the following SSH command wasn’t actually extending the timeout for my source IP address in the relevant clients’ set. So my knock script would rightfully report that SSH was open and then my actual SSH connection would end up failing moments later when it tried to initiate the connection and find the port closed.

The first solution I implemented are the two commented out lines about refreshing the port knock timer, both using the update directive. I’ve left those here in case someone wants this version of the functionality. The way those are written, it will keep updating the timeout as long as any existing SSH connections are open. However, this also means that anyone else coming from the same source IP can attempt to connect for as long as you have your own SSH connections open. This wasn’t exactly what I wanted as I actually liked the fact that SSH was only open for 10 seconds after a successful knock attempt, and then it was closed back down entirely except for any already established connections.

The solution therefore was to simply change the ‘add @clients_ipv[46]’ statements to use update instead. This means that any successful port knock will give you a full 10 seconds instead of whatever small amount of time may have remained from the previous successful port knock, thereby hopefully avoiding this sort of race condition effect I was seeing during my backups.

The remaining pieces to make this work are the knock script itself and altering your SSH configuration to use the knock script correctly. Thankfully, someone else had already figured out the cleanest way to configure SSH to work in this sort of setup. While I’m not a massive fan of needing to run all of my connections through nc for any hosts where I’ve implemented this, I’ve been using similar SSH configurations for years at this point (prior to the ProxyJump directive anyway), so it doesn’t really bother me:

Host tango
  Hostname tango.example.com
  ProxyCommand bash -c '/home/user/bin/knock %h %p 12345 23456 34567 45678; exec nc %h %p'

That is written generically enough that if you use a non-standard SSH port, it should still work. And since my knock script is already checking to ensure the service port is open, I’m skipping the sleep statement used in the source linked above.

And finally, here’s the zsh knock script I’ve concocted:

#!/usr/bin/env zsh

# load module to parse command line arguments
zmodload zsh/zutil
zparseopts -D -E -A opts -- h x

# load module to avoid use of GNU sleep
zmodload zsh/zselect

# enable XTRACE shell option for full debugging output of scripts
if (( ${+opts[-x]} )); then
	set -x
fi

# display short command usage
if [[ -z "${2}" ]] || (( ${+opts[-h]} )); then
	echo "usage: ${0:t} [ -h ] [ -x ] host port [ knock_port ] .." >&2
	echo -e '\n\t-h\tshow this help\n\t-x\tenable shell debugging' >&2
	echo -e '\thost\tdestination host name' >&2
	echo -e '\tport\tdestination service port\n' >&2
	echo -e 'Specifying no knock_port(s) will use 12345 23456 34567 45678 by default.\n' >&2
	exit 1
fi

# define our variables
host="${1}"
port="${2}"
shift 2
knock_ports="${@:-12345 23456 34567 45678}"
attempts=1

# helper function to check whether the service port is actually open
function check_service_port {
	if nc -w1 ${host} ${port} &> /dev/null <& -; then
		exit 0
	fi
}

# check if the service port is already open for some reason
# commented out to avoid race condition and need for additional firewall update rule
#check_service_port

# main loop to open requested service port via port knocking
while [[ ${attempts} -lt 9 ]]; do

	for knock_port in ${=knock_ports}; do
		nc -w1 ${host} ${knock_port} &> /dev/null <& - &
		# increasingly back off on subsequent attempts in case packets are arriving out of order due to high latency
		zselect -t ${attempts}0
	done

	# check now if the service port is open
	check_service_port
	# if not, try again
	((attempts+=1))

done

# all attempts failed, so exit with error
exit 1

And that should be all there is to it! Obviously, this can be used potentially for any service and not just SSH. And depending on traffic conditions between the source and destination, you might need to adjust the number of attempts or the back off timer logic, where zselect is in hundreths of a second, hence adding the zero to the attempt number in the script logic. Similarly, you could increase or decrease the number of required ports in your knock sequence by adjusting the nftables configuration appropriately and passing the requisite number of knock_port arguments to the knock script.

It’s also worth noting that anyone sniffing traffic between your source and destination would potentially be able to discern your port knock sequence which is certainly one of the advantages of something like fwknop. However, again, wanting to avoid the added complexity of running an additional service, this solution is good enough for me, combined with all of my other security mechanisms already in place. Besides, if I started seeing a lot of successful port knocking messages in my logs from IP’s other than my own, I’d also now be aware that something really bad has happened somewhere between my source and destination hosts which would require immediate investigation.

Anyway, hopefully this proves useful to someone else. I’ve certainly enjoyed the total abatement of SSH related log spam as a result of implementing this!

current version of my qemu script

Sat, 29 Aug 2015 14:37:23 +0000

Since I keep posting it other places, but have yet to post it here, I’m including a copy of my current shell script to start qemu:

#!/usr/bin/env zsh

keyboard_id="04d9:0169"
mouse_id="046d:c24a"

keyboard=$(lsusb | grep "${keyboard_id}" | cut -d ' ' -f 2,4 | grep -Eo '[[:digit:]]+' | sed -e 's/^0*//' | xargs -n 2 | sed -e 's/ /./')
mouse=$(lsusb | grep "${mouse_id}" | cut -d ' ' -f 2,4 | grep -Eo '[[:digit:]]+' | sed -e 's/^0*//' | xargs -n 2 | sed -e 's/ /./')

if [[ -z "${keyboard}" || -z "${mouse}" ]]; then
        echo "keyboard (${keyboard}) or mouse (${mouse}) cannot be found; exiting"
        exit 1
fi

for i in {4..7}; do
        echo performance > /sys/devices/system/cpu/cpu${i}/cpufreq/scaling_governor
        #cat /sys/devices/system/cpu/cpu${i}/cpufreq/scaling_governor
done

taskset -ac 4-7 qemu-system-x86_64 \
        -qmp unix:/run/qmp-sock,server,nowait \
        -display none \
        -enable-kvm \
        -M q35,accel=kvm \
        -m 8192 \
        -cpu host,kvm=off \
        -smp 4,sockets=1,cores=4,threads=1 \
        -mem-path /dev/hugepages \
        -rtc base=localtime,driftfix=slew \
        -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root \
        -device vfio-pci,host=02:00.0,bus=root,addr=00.0,multifunction=on,x-vga=on -vga none \
        -device vfio-pci,host=02:00.1,bus=root,addr=00.1 \
        -usb -usbdevice host:${keyboard} -usbdevice host:${mouse} \
        -device virtio-scsi-pci,id=scsi \
        -drive if=none,file=/dev/win/cdrive,format=raw,cache=none,id=win-c -device scsi-hd,drive=win-c \
        -drive if=none,format=raw,file=/dev/sr0,id=blu-ray -device scsi-block,drive=blu-ray \
        -device virtio-net-pci,netdev=net0 -netdev bridge,id=net0,helper=/usr/lib/qemu/qemu-bridge-helper &amp;

sleep 5

#cpuid=0
cpuid=4
for threadpid in $(echo 'query-cpus' | qmp-shell /run/qmp-sock | grep '^(QEMU) {"return":' | sed -e 's/^(QEMU) //' | jq -r '.return[].thread_id'); do
        taskset -p -c ${cpuid} ${threadpid}
        ((cpuid+=1))
done

wait

for i in {4..7}; do
        echo ondemand > /sys/devices/system/cpu/cpu${i}/cpufreq/scaling_governor
        #cat /sys/devices/system/cpu/cpu${i}/cpufreq/scaling_governor
done

The only real change was to automatically search for the keyboard and mouse I want to pass through in case they get unplugged and end up at a different bus address.

even more QEMU-KVM news

Fri, 28 Aug 2015 19:23:09 +0000

It seems like a recent Debian kernel change may have moved the vfio_iommu_type1 feature in the kernel from being statically compiled to a module. This meant I was getting the following when trying to start up qemu:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root,addr=00.0,multifunction=on,x-vga=on: vfio: No available IOMMU models
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 18
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 18
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root,addr=00.0,multifunction=on,x-vga=on: Device initialization failed
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

The esteemed Alex Williamson was quick to reply that this was due to a missing kernel module, vfio_iommu_type1 to be exact. So, add that into /etc/modules and go ahead and modprobe it to avoid a reboot, and you should be good to go.

more QEMU-KVM news

Sat, 01 Aug 2015 10:52:04 +0000

I’m still running a virtualized Windows environment. I just upgraded to Windows 10 Pro using the 2012R2 virtio drivers since native drivers don’t seem to exist yet. All of that is working well.

I ended up skipping the nonsense with irqbalance and simply let it run on every processor. It doesn’t seem to make a huge amount of difference either way. I’m still running with the performance CPU frequency governor as I end up with too much jitter in video and audio playback otherwise. I wonder if running on an Intel processor would be better in this particular area?

I stopped passing through my USB ports directly as I was getting a lot of AMD-Vi error messages from the kernel. Everything was still working, but it was aggravating to have my dmesg full of garbage. So now my /etc/modprobe.d/local.conf looks like:

install vfio_pci /sbin/modprobe --first-time --ignore-install vfio_pci ; \
        /bin/echo 0000:02:00.0 > /sys/bus/pci/devices/0000:02:00.0/driver/unbind ; \
        /bin/echo 10de 1189 > /sys/bus/pci/drivers/vfio-pci/new_id ; \
        /bin/echo 0000:02:00.1 > /sys/bus/pci/devices/0000:02:00.1/driver/unbind ; \
        /bin/echo 10de 0e0a > /sys/bus/pci/drivers/vfio-pci/new_id
options kvm-amd npt=0

And I’ve updated my qemu-system command accordingly:

taskset -ac 4-7 qemu-system-x86_64 \
        -qmp unix:/run/qmp-sock,server,nowait \
        -display none \
        -enable-kvm \
        -M q35,accel=kvm \
        -m 8192 \
        -cpu host,kvm=off \
        -smp 4,sockets=1,cores=4,threads=1 \
        -mem-path /dev/hugepages \
        -rtc base=localtime,driftfix=slew \
        -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root \
        -device vfio-pci,host=02:00.0,bus=root,addr=00.0,multifunction=on,x-vga=on -vga none \
        -device vfio-pci,host=02:00.1,bus=root,addr=00.1 \
        -usb -usbdevice host:10.4 -usbdevice host:10.5 \
        -device virtio-scsi-pci,id=scsi \
        -drive if=none,file=/dev/win/cdrive,format=raw,cache=none,id=win-c -device scsi-hd,drive=win-c \
        -drive if=none,format=raw,file=/dev/sr0,id=blu-ray -device scsi-block,drive=blu-ray \
        -device virtio-net-pci,netdev=net0 -netdev bridge,id=net0,helper=/usr/lib/qemu/qemu-bridge-helper &amp;

I’m using the host:bus.addr format for usbdevice as otherwise I’d be passing through a whole lot of USB ports that would match the vendor_id:product_id format. I also get back my USB3 ports under Linux, should I ever really need them (and can always pass them through to Windows should I need to using this same functionality instead of dealing with the vfio-pci stuff).

I also upgraded my host GPU to a passively cooled GeForce 730 as the 8400 was causing weirdness with my receiver trying to detect audio constantly over the DVI to HDMI converter. This kept interrupting the S/PDIF audio I had coming in from the motherboard. Now everything comes over a proper HDMI connection. However, I was disappointed to discover that apparently there hasn’t been a lot of progress made on passing through lossless, high quality audio formats like TrueHD or DTS-HD MA under Linux. mplayer, mpv, and vlc all seemed to be a bust in this regard and Kodi (formerly XBMC) just crashed my machine due to an unrelated nouveau bug, so I didn’t get to test it any further. I can get normal DTS/AC-3 stuff working over HDMI just fine, but not the fancy stuff. I guess I’ll stick to Windows for playing that stuff back even though it’s all stored on my Linux machine. It would have been nice to get that working directly from Linux.

QEMU, KVM, and GPU passthrough on Debian testing

Wed, 15 Jul 2015 14:55:03 +0000

I decided to take the plunge and try to run everything on one machine. I gutted both of my existing machines and bought a few extra parts. The final configuration ended up using an AMD FX-8350 on an ASRock 970 Extreme4 motherboard with 32GB of RAM in a Fractal Design R5 case. I’ve got a GeForce 8400 acting as the display under Linux and a GeForce 670 GTX being passed through to Windows.

I am using the following extra arguments on my kernel command line:

pci-stub.ids=10de:1189,10de:0e0a rd.driver.pre=pci-stub isolcpus=4-7 nohz=off

The identifiers I’m specifying are for the GPU and HDMI audio on my Geforce 670 so the nouveau driver doesn’t latch onto the card. To further help prevent that situation, the rd.driver.pre statement should load the pci-stub driver as early as possible during the boot process. It’s worth noting I’m using dracut. And finally, isolcpus is basically blocking off those 4 cores to prevent Linux from scheduling any processes on those cores. Along that same line of thinking, I tried to add the following to /etc/default/irqbalance:

IRQBALANCE_BANNED_CPUS=000000f0

but realized the current init.d script that systemd is using to start irqbalance won’t ever pass along that environment variable correctly, so for now, I’m starting irqbalance by hand after boot.

I added these modules to /etc/modules:

vfio
vfio_pci

I added these options to /etc/modprobe.d/local.conf (you might need to remove the continuation characters and make that all one line):

install vfio_pci /sbin/modprobe --first-time --ignore-install vfio_pci ; /bin/echo 0000:02:00.0 &gt; /sys/bus/pci/devices/0000:02:00.0/driver/unbind ; \
        /bin/echo 10de 1189 &gt; /sys/bus/pci/drivers/vfio-pci/new_id ; /bin/echo 0000:02:00.1 &gt; /sys/bus/pci/devices/0000:02:00.1/driver/unbind ; \
        /bin/echo 10de 0e0a &gt; /sys/bus/pci/drivers/vfio-pci/new_id ; /bin/echo 0000:05:00.0 &gt; /sys/bus/pci/devices/0000:05:00.0/driver/unbind ; \
        /bin/echo 1b21 1042 &gt; /sys/bus/pci/drivers/vfio-pci/new_id ; /bin/echo 0000:00:13.0 &gt; /sys/bus/pci/devices/0000:00:13.0/driver/unbind ; \
        /bin/echo 0000:00:13.2 &gt; /sys/bus/pci/devices/0000:00:13.2/driver/unbind ; /bin/echo 1002 4397 &gt; /sys/bus/pci/drivers/vfio-pci/new_id ; \
        /bin/echo 1002 4396 &gt; /sys/bus/pci/drivers/vfio-pci/new_id
options kvm-amd npt=0

So that looks like a mess, but it’s fairly straightforward really. Since I can’t easily pass my device identifiers for USB through the kernel command line, I’m unbinding them individually and rebinding them to the vfio-pci driver. I’m passing through all the USB2 and USB3 devices running the ports on the front of my case. I’m also binding the GPU/audio device to vfio-pci here and specifying an option to KVM which is suppose to help performance on AMD machines. I setup some hugepage reservations and enabled IPv4 forwarding in /etc/sysctl.d/local.conf:

# Set hugetables / hugepages for KVM single guest needing 8GB RAM
vm.nr_hugepages = 4126

# forward traffic
net.ipv4.ip_forward = 1

Since bridging is my network usage model of choice, I needed to change /etc/network/interfaces:

auto lo br0
iface lo inet loopback

iface eth0 inet manual

iface br0 inet dhcp
        bridge_ports eth0
        bridge_stp off
        bridge_waitport 0
        bridge_fd 0

Putting everything I’ve discovered together, I’ve created a shell script. It includes all of the different things that need to happen like setting the cpufreq governor and pinning individual virtual CPU thread process identifiers to their respective physical CPU’s. I’m using zsh as that is my goto shell for all things, but most anything should suffice. The script also depends on the presence of the qmp-shell script available here. You will want both the qmp-shell script itself, and the dependent Python library called qmp.py. Once all of that is in place, here is the final script to start everything:

#!/usr/bin/env zsh

for i in {4..7}; do
        echo performance &gt; /sys/devices/system/cpu/cpu${i}/cpufreq/scaling_governor
        #cat /sys/devices/system/cpu/cpu${i}/cpufreq/scaling_governor
done

taskset -ac 4-7 qemu-system-x86_64 -qmp unix:/run/qmp-sock,server,nowait -display none -enable-kvm -M q35,accel=kvm -m 8192 -cpu host,kvm=off \
        -smp 4,sockets=1,cores=4,threads=1 -mem-path /dev/hugepages -rtc base=localtime -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root \
        -device vfio-pci,host=02:00.0,bus=root,addr=00.0,multifunction=on,x-vga=on -vga none -device vfio-pci,host=02:00.1,bus=root,addr=00.1 \
        -device vfio-pci,host=05:00.0 -device vfio-pci,host=00:13.0 -device vfio-pci,host=00:13.2 -device virtio-scsi-pci,id=scsi \
        -drive if=none,file=/dev/win/cdrive,format=raw,cache=none,id=win-c -device scsi-hd,drive=win-c -drive if=none,file=/dev/win/ddrive,format=raw,cache=none,id=win-d \
        -device scsi-hd,drive=win-d -drive if=none,format=raw,file=/dev/sr0,id=blu-ray -device scsi-block,drive=blu-ray -device virtio-net-pci,netdev=net0 \
        -netdev bridge,id=net0,helper=/usr/lib/qemu/qemu-bridge-helper &amp;

sleep 5

cpuid=4
for threadpid in $(echo 'query-cpus' | qmp-shell /run/qmp-sock | grep '^(QEMU) {"return":' | sed -e 's/^(QEMU) //' | jq -r '.return[].thread_id'); do
        taskset -p -c ${cpuid} ${threadpid}
        ((cpuid+=1))
done

wait

for i in {4..7}; do
        echo ondemand &gt; /sys/devices/system/cpu/cpu${i}/cpufreq/scaling_governor
        #cat /sys/devices/system/cpu/cpu${i}/cpufreq/scaling_governor
done

I force the CPU cores assigned to the VM to run at their maximum frequency for the duration of the guest, after which, they scale back down into their normal on-demand mode. I found this helps to smooth out things a little bit more and helps to provide something approaching a physical machine experience, even though I’m using more power to get there. I’m also using qmp-shell to check the PID’s of the vCPU threads and assigning each of them to individual pCPU’s.

I ended up using the q35 virtual machine layout instead of the default. I’m not positive this matters, but I did end up adding the ioh3420 device later in my testing and it really did seem to improve performance a little bit more. Whether that requires using the q35, I’m not certain. And anyway, once the devices were detected and running under Windows after I first moved from physical to virtual, it wasn’t worth it to me to switch back to the default machine type. I’m also using the legacy SeaBIOS instead of OVMF since I was migrating from physical to virtual and it was too much trouble trying to make a UEFI BIOS work after the fact.

Initially I wasn’t using virtio based hardware, so you’ll possibly need to change that to get up and running and then add in the virtual devices and load the proper virtio drivers. I did run into some weirdness here for a long time where Windows 7 kept crashing trying to install the drivers for either virtio-blk-pci or virtio-scsi-pci. I was using the current testing kernel (linux-image-3.16.0-4-amd64) and never really found a solution. I did end up installing a clean copy of Windows and was able to install the virtio stuff, but this really didn’t help me. I finally ended up installing the latest unstable kernel which is linux-image-4.0.0-2-amd64 and I was finally able to install the virtio stuff without the guest OS crashing. I have no idea if that was the actual fix, but it seemed to be the relevant change.

Another thing that took awhile to figure out was how to properly pass through my Blu-ray drive to Windows so that things like AnyDVD HD worked correctly. I finally stumbled across this PDF which actually included qemu related commands to doing passthrough. It ended up being a simple change from scsi-cd to scsi-block.

I also had to forcibly set the GPU and audio drivers under Windows to use MSI by following these directions. Before doing this, audio was atrocious and video was pretty awful too.

That’s most of it I think. When I originally posted this, I still wasn’t quite happy with the performance of everything. However, in the current incarnation, aside from the possibly excessive power consumption caused by keeping the CPU’s running at full tilt, I’m actually really happy with the performance. Hopefully other people will find this useful too!

DRBD v2

Wed, 08 May 2013 15:12:17 +0000

Previously I had written a fairly lengthy post on creating a cheap SAN using DRBD, iSCSI, and corosync/pacemaker. It was actually the second time we had done this setup at work, having originally done iSCSI LUN’s using logical volumes on top of a single DRBD resource instead of what I described in my last post where we did iSCSI LUN’s which were themselves separate DRBD resources on top of local logical volumes on each node of the cluster. Having run with that for awhile, and added around forty LUN’s, I will say that it is rather slow at migrating from the primary to secondary node and only takes longer as we continue to add new DRBD resources.

Since we’re in the process of setting up a new DRBD cluster, we’ve decided to go back to using the original design of using iSCSI LUN’s using logical volumes on top of one large, single DRBD resource. I’ll also mention that we had some real nightmares using the latest and greatest versions of Pacemaker 1.1.8 in Red Hat Enterprise Linux 6.4, so we’re also pegging our cluster tools at the previous versions of everything which shipped in 6.3. Maybe the 6.4 stuff would have wokred if we were running a cluster in the more tradional Red Hat way (using CMAN).

So now our sl.repo file specifies the 6.3 release:

[scientific-linux]
name=Scientific Linux - $releasever
baseurl=http://ftp.scientificlinux.org/linux/scientific/6.3/$basearch/os/
enabled=1
gpgcheck=0

And we’ve also added a newer version of crmsh which must be installed forcibly from the RPM itself as it overwrites some of the files in the RHEL 6.3 pacemaker packages:

rpm --replacefiles -Uvh http://download.opensuse.org/repositories/network:/ha-clustering/RedHat_RHEL-6/x86_64/crmsh-1.2.5-55.3.x86_64.rpm

We did this specifically to allow use of rsc_template in our cluster which cleans everything up and makes the configuration hilariously simple.

We’ve also cleaned up the corosync configuration a bit by removing /etc/corosync/service.d/pcmk and adding that to the main configuration, as well as making use of the key we generated using corosync-keygen by enabling secauth:

amf {
  mode: disabled
}
 
logging {
  fileline: off
  to_stderr: no
  to_logfile: yes
  to_syslog: no
  logfile: /var/log/cluster/corosync.log
  debug: off
  timestamp: on
  logger_subsys {
    subsys: AMF
    debug: off
    tags: enter|leave|trace1|trace2|trace3|trace4|trace6
  }
}
 
totem {
  version: 2
  token: 10000
  token_retransmits_before_loss_const: 10
  vsftype: none
  secauth: on
  threads: 0
  rrp_mode: active
 
 
  interface {
    ringnumber: 0
    bindnetaddr: 172.16.165.0
    broadcast: yes
    mcastport: 5405
  }
  interface {
    ringnumber: 1
    bindnetaddr: 10.0.0.0
    broadcast: yes
    mcastport: 5405
  }
}

service {
  ver: 1
  name: pacemaker
}

aisexec {
  user: root
  group: root
}
 
corosync {
  user: root
  group: root
}

Other than that, there’s onle one DRBD resource now. And once it’s configured, you shouldn’t ever really need to touch DRBD at all. lvcreate happens only once, and only on the primary storage node. We’ve also learned that corosync-cfgtool -s may not always be the best way to check membership, so you can also check corosync-objctl | grep member.

We also ran across a DRBD related bug in 6.4 which seems to affect this mixed 6.3/6.4 environment as well. We’re still using kmod-drbd84 from El Repo, which is currently at version 8.4.2. Apparently in the shipping version of 8.4.3, they’ve fixed the bug that causes the file /usr/lib/drbd/crm-fence-peer.sh to break things horribly under 6.4 but also seems to work better even using Pacemaker 1.1.7 under 6.3. I recommend grabbing the tarball for 8.4.3 and overwriting the version shipping with 8.4.2. I’m sure as soon as 8.4.3 is packaged and available on El Repo, this won’t be necessary.

You might want to set up a cronjob to run this DRBD verification script once a month or so:

#!/bin/sh

for i in $(drbdsetup show all | grep ^resource | awk '{print $2}' | sed -e 's/^r//'); do
	drbdsetup verify $i
	drbdsetup wait-sync $i
done

echo "DRBD device verification completed"

And maybe run this cluster backup script nightly just so you always have a reference point if something significant changes in your cluster:

#!/usr/bin/env bash

#define some variables
PATH=/bin:/sbin:/usr/bin:/usr/sbin
hour=$(date +"%H%M")
today=$(date +"%Y%m%d")
basedir="/srv/backups/cluster"
daily=$basedir/daily/$today
monthly=$basedir/monthly
lock="/tmp/$(basename $0)"

if test -f $lock; then
	echo "exiting; lockfile $lock exists; please check for existing backup process"
	exit 1
else
	touch $lock
fi

if ! test -d $daily ; then
	mkdir -p $daily
fi

if ! test -d $monthly ; then
	mkdir -p $monthly
fi


# dump and compress both CRM and CIB
crm_dumpfile="crm-$today-$hour.txt.xz"
if ! crm configure show | xz -c &gt; $daily/$crm_dumpfile; then
	echo "something went wrong while dumping CRM on $(hostname -s)"
else
	echo "successfully dumped CRM on $(hostname -s)"
fi

cib_dumpfile="cib-$today-$hour.xml.xz"
if ! cibadmin -Q | xz -c &gt; $daily/$cib_dumpfile; then
	echo "something went wrong while dumping CIB on $(hostname -s)"
else
	echo "successfully dumped CIB on $(hostname -s)"
fi

# keep a monthly copy
if test "x$(date +"%d")" == "x01" ; then
	monthly=$monthly/$today
	mkdir -p $monthly
	cp $daily/$crm_dumpfile $monthly
	cp $daily/$cib_dumpfile $monthly
fi

# remove daily backups after 2 weeks
for dir in $(find "$basedir/daily/" -type d -mtime +14| sort); do
	if test -d "$dir"; then
		echo "removing $dir"
		rm -rf "$dir"
	else
		echo "$dir not found"
	fi
done

# remove monthly backups after 6 months
for dir in $(find "$basedir/monthly/" -type d -mtime +180| sort); do
	if test -d "$dir"; then
		echo "removing $dir"
		rm -rf "$dir"
	else
		echo "$dir not found"
	fi
done

rm -f $lock

And finally, we have the actual cluster configuration itself, more or less straight out of production:

node salt
node pepper
rsc_template lun ocf:heartbeat:iSCSILogicalUnit \
	params target_iqn="iqn.2013-04.net.bitgnome:vh-storage01" additional_parameters="mode_page=8:0:18:0x10:0:0xff:0xff:0:0:0xff:0xff:0xff:0xff:0x80:0x14:0:0:0:0:0:0" \
	op start interval="0" timeout="10" \
	op stop interval="0" timeout="10" \
	op monitor interval="10" timeout="10"
primitive fence-salt stonith:fence_ipmilan \
	params ipaddr="172.16.74.164" passwd="abcd1234" login="laitsadmin" verbose="true" pcmk_host_list="salt" \
	op start interval="0" timeout="20" \
	op stop interval="0" timeout="20"
primitive fence-pepper stonith:fence_ipmilan \
	params ipaddr="172.16.74.165" passwd="abcd1234" login="laitsadmin" verbose="true" pcmk_host_list="pepper" \
	op start interval="0" timeout="20" \
	op stop interval="0" timeout="20"
primitive ip ocf:heartbeat:IPaddr2 \
	params ip="172.16.165.24" cidr_netmask="25" \
	op start interval="0" timeout="20" \
	op stop interval="0" timeout="20" \
	op monitor interval="10" timeout="20"
primitive lun1 @lun \
	params lun="1" path="/dev/vg0/vm-ldap1"
primitive lun2 @lun \
	params lun="2" path="/dev/vg0/vm-test1"
primitive lun3 @lun \
	params lun="3" path="/dev/vg0/vm-mail11"
primitive lun4 @lun \
	params lun="4" path="/dev/vg0/vm-mail2"
primitive lun5 @lun \
	params lun="5" path="/dev/vg0/vm-www1"
primitive lun6 @lun \
	params lun="6" path="/dev/vg0/vm-ldap-slave1"
primitive lun7 @lun \
	params lun="7" path="/dev/vg0/vm-ldap-slave2"
primitive lun8 @lun \
	params lun="8" path="/dev/vg0/vm-ldap-slave3"
primitive lun9 @lun \
	params lun="9" path="/dev/vg0/vm-www2"
primitive lvm_vg0 ocf:heartbeat:LVM \
	params volgrpname="vg0" \
	op start interval="0" timeout="30" \
	op stop interval="0" timeout="30" \
	op monitor interval="10" timeout="30" depth="0"
primitive r0 ocf:linbit:drbd \
	params drbd_resource="r0" \
	op start interval="0" timeout="240" \
	op promote interval="0" timeout="90" \
	op demote interval="0" timeout="90" \
	op notify interval="0" timeout="90" \
	op stop interval="0" timeout="100" \
	op monitor interval="20" role="Slave" timeout="20" \
	op monitor interval="10" role="Master" timeout="20"
primitive tgt ocf:heartbeat:iSCSITarget \
	params iqn="iqn.2013-04.net.bitgnome:vh-storage01" tid="1" allowed_initiators="172.16.165.18 172.16.165.19 172.16.165.20 172.16.165.21" \
	op start interval="0" timeout="10" \
	op stop interval="0" timeout="10" \
	op monitor interval="10" timeout="10"
ms ms-r0 r0 \
	meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
location salt-fencing fence-salt -inf: salt
location pepper-fencing fence-pepper -inf: pepper
colocation drbd-with-tgt inf: ms-r0:Master tgt:Started
colocation ip-with-lun inf: ip lun
colocation lun-with-lvm inf: lun lvm_stor01
colocation lvm-with-drbd inf: lvm_stor01 ms-r0:Master
order drbd-before-lvm inf: ms-r0:promote lvm_stor01:start
order lun-before-ip inf: lun ip
order lvm-before-lun inf: lvm_stor01 lun
order tgt-before-drbd inf: tgt ms-r0
property $id="cib-bootstrap-options" \
	dc-version="1.1.7-6.el6-abcd1234" \
	cluster-infrastructure="openais" \
	expected-quorum-votes="2" \
	no-quorum-policy="ignore" \
	stonith-enabled="true" \
	last-lrm-refresh="1368030674" \
	stonith-action="reboot"
rsc_defaults $id="rsc-options" \
        resource-stickiness="100"

The great part about this configuration is that the constraints are all tied to the rsc_template, so you don’t need to specify new constraints each time you add a new LUN. And because we’re using a template, the actual LUN primitives are as short as possible while still uniquely identifying each unit. It’s quite lovely really.

the hell of Java keystores and existing server certificates

Thu, 08 Nov 2012 13:56:16 +0000

As a semi-conscientious netizen, I feel it’s my duty to post about the insanity of dealing with Java keystore files when you already have X.509 PEM encoded certificates and intermediate CA certificates. I spent multiple hours over the last few days trying to grok this mess and I never want to spend another moment of my life trying to reinvent the wheel when I have to do this again several years from now.

Like most server administrators probably, I have an existing set of signed server certificates along with a bundle of CA signed intermediate certificates, all in X.509 PEM format (those base64 encoded ASCII text files that everyone knows and loves in the Unix world; if you’re using Windows and have PKCS #12 encoded files, you’ll need to look up how to convert them using the openssl command). But now I need to deploy something Java based (often Tomcat applications) which requires a Java keystore file instead of the much saner X.509 PEM format that practically everything that isn’t Java uses without any problems. This is where the insanity starts. And yes, I realize that newer versions of Tomcat can use OpenSSL directly which allows you to use X.509 PEM encoded files directly also, but that wasn’t an option here. And yes, I also realize that you could do some crazy wrapper setup using Apache on the front and the Tomcat application on the back. But that’s ludicrous just to work around how idiotic Java applications are about handling SSL certificates.

Every other piece of Unix software I’ve ever configured expects a server certificate and private key and possibly a single or even multiple intermediate certificates to enable SSL or TLS functionality. Granted, some applications are better about explicitly supporting intermediate certificates. But even ones that don’t almost always allow you to concatenate all of your certificates together (in order from least to most trusted; so, server certificate signed by some intermediate signed by possibly another intermediate signed by a self-signed CA certificate, where the top level CA certificate is normally left off of the chain). The point is, the end client ends up getting said blob and can then check that the last intermediate certificate is signed by a locally trusted top level CA certificate already present on the client’s device.

All of the documentation I could find says to import the intermediate and CA certificates into the keystore using the -trustcacerts option and using different aliases. The problem I was seeing though was that testing the validity of my server’s certificate after I installed the keystore this way using OpenSSL’s s_client always resulted in the server certificate not validating. Looking at s_client with -showcerts enabled, all I was ever getting back from the server during the initial SSL handshake was the lone server certificate without any of the intermediate certificates, unlike any of my other Apache or nginx server where the entire certificate blob was being passed from the server to the client, allowing s_client to verify that the certificate was in fact trusted by my local CA bundle installed as part of my operating system. If you want to try validating your own server’s certificate, use something like:

openssl s_client -CAfile /etc/ssl/certs/ca-certificates.crt -showcerts -connect www.bitgnome.net:443

I finally ran across a post which mentioned keystore as part of the vtcrypt project. This turned out to be the key in making everything work the way I normally expect them to work.

Now before you run off to do the magic below, you will need to convert your PKCS #8 PEM formatted private key into a DER formatted key. You will need to do something like:

openssl pkcs8 -topk8 -nocrypt -outform DER -in server.key -out server.pkcs8

The handy thing about keystore is that it will ingest a standard X.509 PEM encoded certificate file, even when it has multiple certificates present, and spit out that desperately needed Java keystore with an alias that actually has multiple ceritficates present also! I include the magic here for demonstration purposes:

~/vt-crypt-2.1.4/bin/keystore -import -keystore test.jks -storepass changeit -alias tomcat -cert server+intermediate.crt -key server.pkcs8

That’s it! The test.jks keystore doesn’t need to exist. This will create it. Check to make sure that the keystore now contains the correct information:

~/vt-crypt-2.1.4/bin/keystore -list -keystore test.jks -storepass changeit

and you should see your certificate chain starting with your server certificate and ending with your last intermediate certificate. Once I installed the keystore in my application, s_client was able to successfully verify the now complete chain of trust from server certificate to my locally trusted CA root certificate.

pretty VIM colors

Mon, 22 Oct 2012 16:29:57 +0000

This one is more for myself so I don’t forget about it (and I can find it again later). There is a nifty project here that is storing a repository of VIM color settings. It’s a damn slick interface and well worth checking out if you’re a VIM user.

Yep.