Using NixOS for my homelab

TL;DR: You can find my Nix config repository here, or skip to the actual installation writeup. I encourage you to see how I tackled different problems and to reuse any code you find useful. Reach out if you have any questions!

Motivation🔗

In a way, this blog post has effectively taken four years to write. I originally became interested in self-hosting and home servers when the pandemic took hold, and then used every mainstream Linux distribution and configuration management system until I came across NixOS, which incredibly triumphs at being both.

Two years ago, I was already fairly experienced with problem solving in Arch Linux. I used Arch on every laptop, desktop, and server I owned. But I still felt uneasy about the fragility of my setup and the likelihood of forgetting how to configure something again if anything were to break. I had the choice of either finishing my Ansible configuration or commit to learning something new. Eventually I became frustrated with the excessive boilerplating and slowness of Ansible. It was time to take the dive into NixOS.

I actually failed to enter the Nix ecosystem twice, first in 2022 and then later in 2023. Functional programming was too alien for me, and I persuaded myself that NixOS was too gimmicky and non-transferable as a skillset. But the allure of declarative management was too promising to ignore. I decided to try once more at the end of 2023.

This time, I did it! I now configure my M1 MacBook Air, an AMD Ryzen desktop computer, and several ThinkCenter Tiny servers using a single repository. Everything from my dotfiles to my Nextcloud installation config is organized together, which would have been nearly impossible otherwise.

Learning NixOS this time around was easier partially because of the steadily increasing availability of high-quality blog posts, videos, and other documentation resources. The reputation of Nix being too clever for its own good is arguably exaggerated and improves everyday as layman-friendly documentation catches up. Furthermore, the proliferation of Nix flakes means more configurations are being shared publicly.

But I ultimately prevailed because I was much more realistic about starting from the most minimal, barebones configuration I could find on GitHub. From there, I incrementally added complexity without smashing into a brick wall of unfamiliar syntax and stack traces. As long as I didn’t frustrate myself trying to immediately emulate an experienced user’s seemingly overengineered config repository, I could trust my sense of curiosity to provide the necessary endurance to push through.

So why go through the trouble of migrating my servers to NixOS?

The obvious benefits of NixOS were known before I started using Nix:

• Your entire operating system points to a single source of truth embodied in your NixOS configuration. No more guessing where to find and place various program configuration files. No more wondering why something doesn’t run on your machine and trying to figure out what the workaround is.

• Thanks to its declarative design, NixOS lets you abstract large chunks of complexity away from system administration. Want to enable some OS feature that would have taken multiple imperative steps to complete? No problem. If Ansible is designed to make your grocery shopping and cooking easier, NixOS just lets you order your food directly.

• Unlike Ansible, which achieves idempotency through implementation, NixOS achieves it by design. You can be certain that a given Nix configuration on machine can and will reliably produce the same output on another, i.e. deterministic builds and reproducible deployments.

• NixOS is fast: I can provision a new {file,media,home automation} server from scratch in roughly 10 minutes from the installer ISO.

• There are other configuration management & system orchestration tools out there that I haven’t mentioned, but nearly every imperative process has the same overall pitfalls. As a practical example, take setting up Nextcloud on a bare metal Arch Linux server. This is a super involved and tedious process. Even the official all-in-one Nextcloud docker-compose.yml file takes some serious time to inspect through and understand. Meanwhile, a comparable Nix configuration that sets up Nextcloud along with a reverse proxy, automated off-site backups, and Redis caching is much easier to write and maintain. Here’s mine for comparison.

But it is the not-so-obvious benefits keep me interested. Nix flakes, as I would later discover, make it possible to share code with other Nix users universally. It’s like a package.json, but for your operating system and developer environment. This enables you to add useful add-on functionality to your system at almost no marginal complexity. For instance, I use sops-nix to manage my secrets and Home Manager to manage my dotfiles by declaring them as dependencies in my flake.nix file.

You can even run programs, such as a linter, directly from your command line without actually having installed anything first.

nix run git+https://git.peppe.rs/languages/statix -- --help

NixOS also lets you easily achieve some very unorthodox system configurations. You can run your computer on tmpfs, since NixOS only needs /boot and /nix to boot up and rebuild your entire system configuration from scratch everytime your computer turns on. Everything can be tracked in your VCS, and you’ll easily know the entire surface area of your device with great precision.¹

If you found this post interesting and you’re ready to take the leap into NixOS, feel free to check out my repository.

Installing NixOS on bare metal servers🔗

Okay, here comes the fun part! I’ll walk you through how I install NixOS on the three servers pictured above.

My goal with the three servers was the following setup:

Creating a custom ISO🔗

Since I velcroed my server’s power cables in quite an unconvenient way, I couldn’t be bothered to plug them into a monitor and keyboard for the installation. Luckily, NixOS makes it very easy to generate a custom ISO with your SSH public key so you can handle installation remotely.

On a computer that had nix installed,² I created flake.nix in an empty directory.

{
  description = "custom nixos iso";
  inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";

  outputs = { self, nixpkgs }: {
    nixosConfigurations = {
      exampleIso = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux";
        modules = [
          "${nixpkgs}/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix"

          ({
            users.users.nixos = {
              openssh.authorizedKeys.keys = [
                "ssh-ed25519 <YOUR PUBLIC KEY HERE>"
              ];
            };
          })
        ];
      };
    };
  };
}

Then I ran the following to generate a custom ISO with an SSH public key already embedded:

$ git init
$ git add flake.nix
$ nix build .#nixosConfigurations.exampleIso.config.system.build.isoImage

The ISO file will be generated in a directory called result.

Remotely entering NixOS installer🔗

I use Ventoy on a flash drive to store multiple ISO files on the same bootable USB drive. This is super convenient for using a single USB drive for installing multiple OSes. My flash drive has the following structure:

.
├── ISO
│   ├── Win11_22H2_English_x64v1.iso
│   ├── archlinux-2024.03.01-x86_64.iso
│   ├── nixos-custom-x86_64-linux.iso
│   ├── nixos-gnome-23.11.3496.a77ab169a83a-x86_64-linux.iso
│   └── nixos-minimal-23.11.3496.a77ab169a83a-x86_64-linux.iso
└── ventoy
    └── ventoy.json

Notice how I created ventoy/ventoy.json to automatically load the newly generated NixOS ISO:

{
  "control": [
    {
      "VTOY_DEFAULT_SEARCH_ROOT": "/ISO"
    },
    {
      "VTOY_MENU_TIMEOUT": "5"
    },
    {
      "VTOY_DEFAULT_IMAGE": "/ISO/nixos-custom-x86_64-linux.iso"
    },
    {
      "VTOY_SECONDARY_BOOT_MENU": "1"
    },
    {
      "VTOY_SECONDARY_TIMEOUT": "5"
    }
  ]
}

I plugged my USB drive into the first server and rebooted it. After SSH’ing in, I was greeted not by my familiar Arch Linux environment, but rather than NixOS installer ISO. Nice!

Running my NixOS install script🔗

I created a install script to partition the NVMe drive that would serve as the main filesystem for the server, set up full disk encryption and prompt me for a password, mount the new partitions, and generate a new age public key that I would use with sops-nix. Here is an excerpt of it:

# Define disk
DISK="/dev/nvme0n1"
DISK_BOOT_PARTITION="/dev/nvme0n1p1"
DISK_NIX_PARTITION="/dev/nvme0n1p2"

# Undo any previous changes if applicable
set +e
umount -R /mnt
cryptsetup close cryptroot
set -e

# Partitioning disk
parted $DISK -- mklabel gpt
parted $DISK -- mkpart ESP fat32 1MiB 512MiB
parted $DISK -- set 1 boot on
parted $DISK -- mkpart Nix 512MiB 100%

# Setting up encryption
cryptsetup -q -v luksFormat $DISK_NIX_PARTITION
cryptsetup -q -v open $DISK_NIX_PARTITION cryptroot

# Creating filesystems
mkfs.fat -F32 -n boot $DISK_BOOT_PARTITION
mkfs.ext4 -F -L nix -m 0 /dev/mapper/cryptroot

# Let mkfs catch its breath
sleep 2

# Mounting filesystems
mount -t tmpfs none /mnt
mkdir -pv /mnt/{boot,nix,etc/ssh,var/{lib,log}}
mount /dev/disk/by-label/boot /mnt/boot
mount /dev/disk/by-label/nix /mnt/nix
mkdir -pv /mnt/nix/{secret/initrd,persist/{etc/ssh,var/{lib,log}}}
chmod 0700 /mnt/nix/secret
mount -o bind /mnt/nix/persist/var/log /mnt/var/log

# Generating initrd SSH host key
ssh-keygen -t ed25519 -N "" -C "" -f /mnt/nix/secret/initrd/ssh_host_ed25519_key

# Creating public age key for sops-nix
sudo nix-shell --extra-experimental-features flakes -p ssh-to-age --run 'cat /mnt/nix/secret/initrd/ssh_host_ed25519_key.pub | ssh-to-age'

I took the resulting generated public key and added it to the .sops.yaml file at the root of my repository. After synchronizing the new keys by running for file in secrets/*; do sops updatekeys "$file"; done and committing these changes to GitHub, I was ready to install NixOS on the new server.

Drawing the rest of the owl🔗

I could control what type of setup I wanted to install by adjusting the hostname parameter at the end of my flake install command to svr1chng, svr2chng, or svr3chng.

$ sudo nixos-install --no-root-passwd --root /mnt --flake github:eh8/chenglab#hostname

After a few minutes, the installation process was completed with no issues and I could reboot the server without the USB drive.

The server spawned an SSH daemon on boot to receive the LUKS decryption key. To unlock the server, I logged into the root account for the designated IP address:

$ ssh root@<SERVER IP>
Last login: Sun Mar 24 05:24:28 2024 from 10.0.0.1
Passphrase for /dev/nvme0n1p2:
Waiting 10 seconds for LUKS to request a passphrase........Connection to <SERVER IP> closed by remote host.
Connection to <SERVER IP> closed.

The server then proceeded with the bootup sequence normally. Using Tailscale, I logged into my server to a complete, freshly provisioned server.³

$ ssh svr1chng
# tada!

Anatomy of my configuration🔗

Modular configuration🔗

I began by using taking the minimal example from a GitHub repository conveniently named nix-starter-configs by Misterio77.⁴ The standard example was too confusing for me at first.

I ended up creating my flake.nix by repurposing a stanza for each machine that I would install Nix on. This is how I define the nixosConfigurations for one of my servers.

...

nixosConfigurations = {
  svr1chng = nixpkgs.lib.nixosSystem {
    specialArgs = {inherit inputs outputs;};
    modules = [./machines/svr1chng/configuration.nix];
  };
};

...

Each machine entry in flake.nix links to a folder that contains each machine’s configuration.nix and hardware-configuration.nix.

Even without much of a clear idea on how to use Nix or structure it, I had a design goal that I should separate each distinct function that I wanted my homelab to do into its own Nix file. So since I manage several machines that share parts of a common configuration, any configuration.nix file is just a combination of imported files.

{
  inputs,
  outputs,
  ...
}: {
  imports = [
    inputs.impermanence.nixosModules.impermanence
    inputs.home-manager.nixosModules.home-manager

    ./hardware-configuration.nix

    ./../../modules/nixos/base.nix
    ./../../modules/nixos/packages.nix
    ./../../modules/nixos/remote-unlock.nix
    ./../../modules/nixos/auto-update.nix

    ./../../services/tailscale.nix
    ./../../services/netdata.nix
    ./../../services/nextcloud.nix
  ];

  home-manager = {
    extraSpecialArgs = {inherit inputs outputs;};
    useGlobalPkgs = true;
    useUserPackages = true;
    users = {
      eh8 = {
        imports = [
          ./../../modules/home-manager/base.nix
          ./../../modules/home-manager/packages.nix
          ./../../modules/home-manager/zsh.nix
        ];
      };
    };
  };

  networking.hostName = "svr1chng";
}

There is technically a more idiomatic way to go about doing this. Importing Nix files works fine, but the official Nix repository activates configurations by creating modules that let you activate via config.<your module>.enable = true. This nice person on Reddit kindly included some examples of how this works.

For now, this is the structure that keeps my repository straightforward, scalable across multiple machines, and economical as it concerns boilerplate code. In my opinion, it’s also not overwhelmingly intimidating to new Nix users. This structure means that my Nix files rarely exceed 100 lines and I don’t need to nest directories more than two layers deep.

Remote initrd unlocking🔗

Recall from the above installation script that we generated an initrd host key.

{config, ...}: {
  boot.kernelParams = ["ip=dhcp"];
  boot.initrd.network = {
    enable = true;
    ssh = {
      enable = true;
      shell = "/bin/cryptsetup-askpass";
      authorizedKeys = config.users.users.eh8.openssh.authorizedKeys.keys;
      hostKeys = ["/nix/secret/initrd/ssh_host_ed25519_key"];
    };
  };
}

Beware, using boot.kernelParams = ["ip=dhcp"]; means you must be able to supply an IP address to the machine during bootup, or else it won’t boot.

I had to modify my hardware-configuration.nix file to ensure my ethernet driver was available to the initrd. You can see which driver your machine uses by running readlink /sys/class/net/<YOUR NETWORK INTERFACE>/device/driver when you’re in the NixOS installer ISO. I discovered that for my servers, it’s e1000e.

{
  config,
  lib,
  modulesPath,
  ...
}: {
  imports = [
    (modulesPath + "/installer/scan/not-detected.nix")
  ];

  boot = {
    initrd = {
      # `readlink /sys/class/net/enp0s31f6/device/driver` indicates "e1000e" is the ethernet driver for this device
      availableKernelModules = ["nvme" "xhci_pci" "ahci" "usb_storage" "sd_mod" "e1000e"];
      luks = {
        reusePassphrases = true;
        devices = {
          "cryptroot" = {
            device = "/dev/nvme0n1p2";
            allowDiscards = true;
          };
          "fun" = {
            device = "/dev/sda1";
          };
        };
      };
    };
  };

  fileSystems = {
    "/" = {
      device = "none";
      fsType = "tmpfs";
      options = ["defaults" "size=2G" "mode=0755"];
    };
    "/boot" = {
      device = "/dev/disk/by-label/boot";
      fsType = "vfat";
      options = ["umask=0077"];
    };
    "/nix" = {
      device = "/dev/disk/by-label/nix";
      fsType = "ext4";
    };
    "/fun" = {
      device = "/dev/disk/by-label/fun";
      fsType = "ext4";
    };
  };

  networking.useDHCP = lib.mkDefault true;
  nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
  hardware.cpu.intel.updateMicrocode = lib.mkDefault config.hardware.enableRedistributableFirmware;
}

Unattended upgrades🔗

I also set up a GitHub action bot on my repository that updates my flake.lock file daily. I wrote a file called auto-update.nix that, as you guessed, can automatically update a NixOS system by reading from the tip of my repository’s commit tree.

This is the GitHub action YAML:

---
# inspo: https://github.com/reckenrode/nixos-configs/blob/main/.github/workflows/main.yml
name: Bump flake.lock
on:
  schedule:
    - cron: 0 6 * * *
  workflow_dispatch: null
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: cachix/install-nix-action@v26
      - run: nix flake update
      - uses: stefanzweifel/git-auto-commit-action@v5
        with:
          commit_message: "chore: bump flake.lock"
          commit_user_name: Flake Bot
          commit_options: --no-verify --signoff
          commit_author: Flake Bot [email protected]>
          branch: main
          file_pattern: flake.lock
          skip_dirty_check: false
          skip_fetch: true

And here’s the Nix config that manages unattended upgrades:

{
  # inspo: https://github.com/reckenrode/nixos-configs/blob/main/hosts/meteion/configuration.nix
  system.autoUpgrade = {
    enable = true;
    dates = "*-*-* 07:00:00";
    randomizedDelaySec = "1h";
    flake = "github:eh8/chenglab";
  };
}

Acknowledgments🔗

For beginners, this was an excellent resource to learn more about NixOS and flakes.

Manuel Hutter’s blog post NixOS on Hetzner Dedicated was both timely and useful. It inspired large parts of my own install script.