My Nix(OS) playground and configs
  • Nix 67.2%
  • Shell 16.9%
  • QML 7.2%
  • Lua 4.8%
  • Just 2.7%
  • Other 1.2%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Julius Zeidler b7938a4c51
All checks were successful
Deploy gate / build servers → deploy (push) Successful in 8m35s
Security scan / Generate SBOM (push) Successful in 15m48s
Security scan / vulnix — CVE scan (push) Successful in 24m39s
feat(hetzner-robot-1): nightly backup of code.zeidler.dev
2026-10-27 23:48:54 +01:00
.claude chore(overlays): drop overlays obsoleted by nixpkgs 2026-10-12 16:45:16 +02:00
.forgejo/workflows ci: fix security-scan, make update-flake a nightly all-host build check 2026-10-25 18:09:00 +01:00
ci ci: fix security-scan, make update-flake a nightly all-host build check 2026-10-25 18:09:00 +01:00
configs fix(hyprland): disable big-rig's MSI on tux 2026-10-24 15:40:26 +02:00
docs/wiki feat(reticulum): add LoRa mesh stack; drop herald and connect-iq-sdk-manager-cli 2026-10-25 18:06:23 +01:00
hosts feat(hetzner-robot-1): nightly backup of code.zeidler.dev 2026-10-27 23:48:54 +01:00
installer fix(installer): release LVM/swap on target disk before wiping 2026-10-11 17:04:25 +02:00
modules feat(hetzner-robot-1): code.zeidler.dev stack, admin SSH only on 2342 2026-10-27 22:48:17 +01:00
overlays chore(overlays): drop overlays obsoleted by nixpkgs 2026-10-12 16:45:16 +02:00
pkgs feat(reticulum): add LoRa mesh stack; drop herald and connect-iq-sdk-manager-cli 2026-10-25 18:06:23 +01:00
scripts fix(mail): correct default bitwarden folder name to lowercase email 2026-07-05 14:37:27 +02:00
secrets feat(pangolin): enable tux's client tunnel 2026-09-17 21:02:11 +02:00
.gitignore chore: ignore Claude Code worktrees 2026-10-24 12:34:34 +02:00
.sops.yaml feat(pangolin): add Pangolin site tunnel client, secrets via sops-nix 2026-09-17 19:22:29 +02:00
CLAUDE.md feat(reticulum): add LoRa mesh stack; drop herald and connect-iq-sdk-manager-cli 2026-10-25 18:06:23 +01:00
flake.lock chore(flake): roll nixpkgs back to 7fd4004, drop pacu 2026-10-25 02:40:38 +02:00
flake.nix feat(reticulum): add LoRa mesh stack; drop herald and connect-iq-sdk-manager-cli 2026-10-25 18:06:23 +01:00
justfile fix(just): guard NIX_SSHOPTS under set -u in deploy reboot 2026-10-25 02:40:32 +02:00
jz@chaos.li.pub.asc add pub key 2026-07-03 22:56:26 +02:00
README.md feat(hetzner-robot-1): nightly backup of code.zeidler.dev 2026-10-27 23:48:54 +01:00
tools.md feat(reticulum): add LoRa mesh stack; drop herald and connect-iq-sdk-manager-cli 2026-10-25 18:06:23 +01:00

nixconfig — home-manager flake

First-time setup

1. Install Nix (Determinate Systems installer)

curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install

This installs Nix with flakes enabled by default. Restart your shell after.

2. Install home-manager

nix run home-manager/master -- init --switch

Or, once your flake is ready, pick your configuration:

Machine Command
tux — NixOS bare-metal (UEFI), x86_64 sudo nixos-rebuild switch --flake ~/Projects/nix/nixconfigs#tux (or hms)
big-rig — NixOS bare-metal (UEFI), x86_64, NVIDIA gaming desktop sudo nixos-rebuild switch --flake ~/Projects/nix/nixconfigs#big-rig (or hms) — install via the installer ISO (interactive) or its own unattended ISO
homelab-1/2/3 — headless NUC k3s cluster, x86_64 No console needed — apply from another machine: just deploy homelab-1 (or sudo nixos-rebuild switch --flake ~/Projects/nix/nixconfigs#homelab-1 if SSH'd in). Install via the homelab-n unattended ISO
hetzner-robot-1 — Hetzner Robot dedicated server (legacy BIOS), x86_64 just deploy hetzner-robot-1 65.21.79.239 (just reboot=true deploy … reboots into the new config, e.g. for a new kernel). Installed remotely with nixos-anywhere
NixOS VM, x86_64 sudo nixos-rebuild switch --flake ~/Projects/nix/nixconfigs#nixos-vm
Other NixOS bare-metal (UEFI), x86_64 Install via the installer ISO → finish-setup.sh; then sudo nixos-rebuild switch --flake ~/Projects/nix/nixconfigs#<hostname> (or hms)
MacBook M2 (aarch64) home-manager switch --flake ~/Projects/nix/nixconfigs#macbook-silicon

MacBook Intel is temporarily dropped from the flake — nixpkgs-unstable 26.11 dropped x86_64-darwin support entirely. hosts/macbook-intel/home.nix is left in place; it's expected to become a NixOS host later, same as big-rig.

3. Activate for the first time

# NixOS host (tux, big-rig):
sudo nixos-rebuild switch --flake ~/Projects/nix/nixconfigs#tux
# macOS host (home-manager only):
home-manager switch --flake ~/Projects/nix/nixconfigs#macbook-silicon

4. Subsequent updates

sudo nixos-rebuild switch --flake ~/Projects/nix/nixconfigs#tux
# or add a shell alias: alias hms="sudo nixos-rebuild switch --flake ~/Projects/nix/nixconfigs#tux"

One-time setup — logiops

Fully declarative via modules/nixos/logiops.nix — the service and config are managed by NixOS on every NixOS host (tux, big-rig). No manual steps needed.

Historical note: while big-rig ran CachyOS/Arch (home-manager-only, before its 2026-08-30 conversion to NixOS), logiops needed a manual pacman -S logiops + systemctl edit logid pointing at a home-manager-deployed ~/.config/logid.cfg. That no longer applies now that big-rig is NixOS — noted here only in case another Arch host is ever added.


NixOS VM (nixos-vm)

nixos-vm is a dev/test VM built from the same modules/nixos/common.nix and modules/nixos/desktop.nix as real machines, differing only in its bootloader — BIOS GRUB on a virtio disk (/dev/vda) instead of UEFI systemd-boot. Home-manager is a NixOS module, so one command applies system + dotfiles:

sudo nixos-rebuild switch --flake ~/Projects/nix/nixconfigs#nixos-vm   # or: hms

The system-level Hyprland/Wayland pieces (compositor, pipewire, polkit, tty1 autologin, fish login shell) live in modules/nixos/desktop.nix; on tty1 login modules/fish.nix execs start-hyprland (custom.hyprlandLaunchCmd), the launcher binary nixpkgs' hyprland package ships (sets up the systemd user session before starting the compositor — running Hyprland directly prints a warning).


NixOS machines — the template model (nixos-template)

Every physical NixOS machine is provisioned from one generic host, nixos-template, then "graduated" into its own hosts/<hostname>/ entry on first boot. So a single installer image serves many machines (AMD/Intel CPUs, with or without NVIDIA), all sharing:

  • modules/nixos/common.nix — WiFi (wpa_supplicant), podman (docker CLI shim, no daemon; docker-compose as the podman compose provider), docker-sbx (sbx, pkgs/docker-sbx/; big-rig's interim CachyOS + standalone home-manager bridge gets it from hosts/big-rig/home.nix instead), tuicr (review AI-generated diffs like a GitHub PR, from the terminal), openssh. Deliberately role-agnostic — every real NixOS machine imports it, desktop and headless server alike, so it carries only things every machine wants. Steam, Bluetooth (blueman), and the full libvirt/QEMU VM stack (libvirtd + virt-manager) live in modules/nixos/desktop-common.nix instead — shared by desktop.nix (Hyprland: tux, big-rig, nixos-template, nixos-vm) and gnome.nix (latitude-7212, which deliberately doesn't import desktop.nix — kept self-contained so the Hyprland path stays untouched; Bluetooth there isn't optional, it's that tablet's only keyboard). Neither gets pulled into the headless homelab-n k3s cluster (modules/nixos/server.nix imports neither).
  • generation cleanup — boot.loader.systemd-boot.configurationLimit = 3 keeps only the last 3 system generations (older ones pruned from the boot menu and /nix/var/nix/profiles/system on each rebuild); nix.gc runs weekly to reclaim the store space that frees up once they're pruned. nix.settings.min-free/max-free (5 GiB / 20 GiB) back that up in real time — the daemon runs GC mid-build if free space drops too low, instead of a large download (e.g. tor-browser) failing with "No space left on device" between weekly runs.
  • modules/nixos/desktop.nix — Hyprland, pipewire, Flatpak, Thunar file manager (programs.thunar + archive/volman plugins, xarchiver as the archive plugin's backend for right-click extract/compress incl. .rar, gvfs for trash/mounting, tumbler for thumbnails), tty1 autologin, upower, plus desktop-common.nix (Steam, Bluetooth, libvirtd/virt-manager — see above). sbx login needs a Secret Service provider (services.gnome.gnome-keyring.enable) — since tty1 autologin skips the password prompt, set the first login keyring's password to blank so it unlocks without a passphrase from then on.
  • declarative Flatpak via nix-flatpak (modules/nixos/desktop.nix) — Flathub remote + Zen Browser (app.zen_browser.zen), auto-updated on rebuild. (Bitwarden used to live here too; moved to the native pkgs.bitwarden-desktop in each host's home.nix — the Flatpak sandbox can't expose the udev HID metadata Nitrokey/FIDO2 unlock needs, see the limitations table below.) Dark mode for these needs xdg-desktop-portal-gtk in home-manager's own xdg.portal.extraPortals (modules/home/desktop/hyprland.nix) — Hyprland's home-manager module auto-enables its own portal dir via NIX_XDG_DESKTOP_PORTAL_DIR, which otherwise shadows the system portals and hides the Settings interface Flatpak apps query. Zen ships no static xdg-download access, so downloads otherwise fall through the documents portal into /run/user/$UID/doc/… instead of ~/Downloads; a services.flatpak.overrides entry grants xdg-download:rw so files land in ~/Downloads.
  • kanata keyboard remapper (modules/nixos/kanata.nix, imported per-host) — GACS home row mods + Caps Lock as tap-Esc/hold-Ctrl; config in configs/kanata.kbd (override per-host via custom.kanataConfig)
  • Pangolin tunnel (modules/nixos/pangolin.nix, opt-in via customPangolin.enable + mode) — connects a host to the self-hosted Pangolin server, as either a site (Newt: exposes this host's local resources outward — siteId) or a client (Olm: reaches resources exposed by other sites, exposes nothing of its own — clientId, default mode). Most personal machines want "client". The secret is never in the Nix store or a CLI arg (readable by any local user via ps//proc); it's fed to the systemd service (pangolin-site/pangolin-client, depending on mode) via a root-only EnvironmentFile sourced from sops-nix (secrets/<hostname>.yaml, listed as a recipient in .sops.yaml). Each host decrypts its own file with its own SSH host key — no separate age key to manage — via nix run nixpkgs#ssh-to-age -- -i /etc/ssh/ssh_host_ed25519_key.pub. A live connected/connecting/ failed indicator ships in waybar (configs/waybar/pangolin.sh, custom/pangolin module — detects whichever of the two services is configured; click tails its journal). Add a new host by appending its age key to .sops.yaml, encrypting secrets/<hostname>.yaml (sops --encrypt --in-place), and setting sops.defaultSopsFile + customPangolin in its hosts/<hostname>/default.nix. A site ID and a client ID are different credential types from the Pangolin admin panel — generating one doesn't give you the other.
  • per-host home profile (hosts/<hostname>/home.nix) — imports modules/home/default.nix and modules/home/desktop; host-specific packages added inline. modules/home/default.nix also carries the archive set every host gets: zip/unzip, unar (extracts .rar incl. RAR5, .7z, .zip, .tar.* — LGPL, so no unfree unRAR needed; lsar lists contents) and _7zz (official 7-Zip as 7zz, built without its unfree RAR code)
  • hyprsunset blue-light filter (started by hyprland.lua, scheduled in modules/home/desktop/hyprland.nix → hyprsunset.conf): identity by day, 4000 K from 20:30
  • XDG user dirs (modules/home/desktop/default.nix → xdg.userDirs) — writes ~/.config/user-dirs.dirs with the standard paths and creates the folders, so XDG_DOWNLOAD_DIR (~/Downloads, where the Zen Flatpak lands downloads) etc. are explicit rather than relying on the spec fallback
  • home-manager wired as a NixOS module, so nixos-install/nixos-rebuild applies system and dotfiles together (hms runs nixos-rebuild switch on NixOS hosts)

Hardware differences are handled per-host: CPU microcode (AMD/Intel) is written into each machine's hardware-configuration.nix by nixos-generate-config. NVIDIA env vars live in the host's home.nix via custom.hyprlandExtraLua. NixOS hosts are listed explicitly in flake.nix (nixosHosts); add a new machine by creating hosts/<name>/ and appending its name there.

Install → graduate

  1. Boot the installer ISO and run sudo bootstrap. It installs nixos-template, asks for this machine's hostname, and drops ~/finish-setup.sh (with the hostname baked in). First boot already has your dotfiles (home-manager is a NixOS module).
  2. Log in as jz and run:
    ./finish-setup.sh
    
    It clones the repo to ~/Projects/nix/nixconfigs, creates hosts/<hostname>/ — the real hardware-configuration.nix from /etc/nixos/ plus a generated default.nix — commits it, then nixos-rebuild switch --flake ~/Projects/nix/nixconfigs#<hostname>.
  3. Review and git push when happy.

From then on manage the machine with sudo nixos-rebuild switch --flake ~/Projects/nix/nixconfigs#<hostname> (or hms). The nixos-template install artifact under /etc/nixconfig is leftover and can be deleted.

hosts/nixos-template/hardware-configuration.nix is a placeholder; each real host gets the genuine one via finish-setup.sh (flakes only see git-tracked files, so it's committed). For a BIOS machine, swap the boot.loader.systemd-boot/efi lines for boot.loader.grub.{enable,device} in the generated host config.

Tablet hosts (latitude-7212)

Dell Latitude 7212 Rugged Extreme — an Intel Kaby Lake (i5-7300U) tablet used touch-first (optional Bluetooth keyboard). All hardware is mainline (i915, iwlwifi 8265, btusb, Atmel maXTouch, snd_hda); the single M.2 slot takes SATA or NVMe, so regenerate hardware-configuration.nix on the machine after the SSD is installed. Unlike the other NixOS hosts it runs GNOME, not Hyprland — GNOME has the most mature touch story (auto on-screen keyboard, gestures, native rotation):

  • Intel, not AMD — kvm-intel + hardware.cpu.intel.updateMicrocode (auto from nixos-generate-config); does not import npu.nix (AMD XDNA2-only, would pin linuxPackages_latest).
  • modules/nixos/gnome.nix (in place of desktop.nix) — GNOME on Wayland via GDM, plus the compositor-agnostic desktop services (audio, portals, fonts, Flatpak, keyring) and the same GUI-app Flatpak set as tux (Zen, Discord, Signal, Betterbird, OnlyOffice, Flatseal) plus native bitwarden-desktop in home.nix (see limitations table). Self-contained so the tux/Hyprland desktop.nix is untouched. Autologin is on (unencrypted install). GNOME's built-in OSK auto-appears in text fields — no wvkbd/hardware-button plumbing needed.
  • modules/nixos/tablet.nix — just the iio accelerometer (GNOME reads it for native auto-rotation) and brightnessctl. The old wvkbd + P1-button triggerhappy toggle and modules/home/touch.nix were removed when the host moved to GNOME (its OSK replaces them). cosmic.nix was deleted.
  • hosts/latitude-7212/home.nix — mirrors tux's software minus the Hyprland/waybar-specific bits and the terminal mail stack (GUI apps incl. mail come from Flatpak). Also drops several TUIs that are poor fits for a touch device — aria2tui, bookokrat (terminal e-reader), eilmeldung (news ticker) — which also trims what the tablet builds from source (bookokrat has no binary cache; eilmeldung now ships prebuilt via nixpkgs unstable but is still dropped as a poor touch fit). Sets GNOME's on-screen keyboard on and a dark UI via dconf.settings.
  • modules/nixos/nvme-stable.nix — the machine's WD Black NVMe SSD hard-freezes under APST (Autonomous Power State Transition): the drive enters its deepest sleep state and never wakes, so the next I/O times out and the box hangs (nvme … I/O … timeout / controller … reset in the log; observed even while idle at a shell). It sets nvme_core.default_ps_max_latency_us via the option stabilityWorkarounds.nvme.apstMaxLatencyUs and is shared — imported by this host and by both installer ISOs (installer/iso.nix, installer/unattended.nix), because the live installer runs on the same WD Black and froze on it mid-install. The value defaults to 0 (APST fully off — the installers keep this, since a one-shot install can't roll back); the host relaxes it to 5500, keeping the shallow battery-saving states and forbidding only the deepest, hanging one. Distinct from the pcie_aspm=off in iwlwifi-stable.nix — that is PCIe-link power management, whereas APST is the drive's own internal power-state machine. fwupd (for an eventual WD firmware fix via LVFS) is enabled on the host itself, not in the module, to keep it off the ISOs.
  • modules/nixos/thermald.nix — with APST fixed, the install got far enough to surface a second freeze: mce: [Hardware Error]: Machine check events logged, a hard lock under the install's sustained CPU load (this fanless rugged tablet's Kaby Lake overheats and machine-checks). thermald (Intel Thermal Daemon) throttles the CPU proactively to keep it below its fault point. Intel-only and idle on other hardware, so it's shared by this host and both installer ISOs (the live installer cooked the CPU too). Note this only mitigates thermal MCEs — if machine-checks persist, suspect cooling (dust/thermal paste) or failing RAM/CPU (decode via pstore + the sampler's temps).
  • modules/nixos/no-suspend.nix — this tablet cannot resume from suspend, so suspend is disabled outright. Exhaustively confirmed on-device: under s2idle the power button never wakes it (s2idle resumes on a live IRQ, but /proc/acpi/wakeup arms the power button PBTN only for S3), and under deep/S3 it is equally dead — after systemctl suspend the machine is completely unreachable over Tailscale (via power and Home button), i.e. genuinely hung, not just a dark panel. No wake path exists in any state. The module masks the sleep targets (systemd.targets.{sleep,suspend, hibernate,hybrid-sleep}.enable = false) so nothing — manual systemctl suspend, the GNOME power menu, idle timeout, or critical battery — can suspend the box into a hard-reset; they fail cleanly instead. The host dconf also sets idle auto-suspend to nothing so GNOME never attempts it. The screen still blanks for battery saving. Host-only (not on the installer ISOs — they never suspend). If a firmware/kernel update ever fixes resume, drop the module and revisit mem_sleep_default.
  • modules/nixos/display-stable.nix — a black screen isn't proof of a freeze. Two i915/console causes leave the machine running with the panel dark: the Linux VT blanks after ~10 min of no keyboard input (consoleblank, default 600 s — and an unattended install has no input, so it goes dark mid-run while still installing), and Kaby Lake i915 Panel Self Refresh / framebuffer compression can spontaneously black the display. Always sets consoleblank=0 (needed by the installer VT, a no-op on the Wayland host); the installer also runs setterm --blank 0 --powerdown 0. The i915 enable_psr=0 enable_fbc=0 params are gated behind stabilityWorkarounds.display.disableI915Psr (default true → PSR/FBC off everywhere). Re-enabling them on the host was tried for panel battery but froze the GNOME session a few seconds after login (PSR holds a stale self-refreshed frame on this Kaby Lake panel → the desktop looks hung), so the host keeps the default and PSR/FBC stay off. Shared by the host and both installer ISOs. Diagnostic tip: if the screen goes black, don't power off — press Caps Lock (LED toggles → kernel alive → display issue, not a freeze) and wait; the install may finish blind.
  • modules/nixos/tlp.nix — battery tuning. The host previously had zero CPU/PCIe power management (only thermald, which throttles reactively near the thermal trip point, not proactively for battery). Enables TLP (powersave governor + power/balance_power EPP, turbo boost off on battery, USB autosuspend, audio power-save) and disables power-profiles-daemon (GNOME's mkDefault true, but it and TLP share the same sysfs knobs and can't run together — TLP's finer-grained tuning wins over ppd's quick-settings toggle). Deliberately leaves PCIe ASPM and NVMe APST untouched — this host already fought hard-freezes on both (iwlwifi-stable.nix, nvme-stable.nix) and forces wifi power_save off, so TLP's wifi/PCIe knobs are explicitly set to match those workarounds instead of fighting them. hosts/latitude-7212/home.nix dconf pairs with this: enable-animations = false (cuts shell-compositor GPU work) and a shorter idle-delay (120 s, down from GNOME's 300 s default) plus idle-dim — the screen panel is the single largest power draw on this tablet, larger than CPU idle draw, and nothing here can suspend anyway (see no-suspend.nix) so a short blank timeout costs nothing but a tap to wake.

Battery tuning (tux)

modules/nixos/tlp-tux.nix — tux is an AMD Ryzen AI laptop (XDNA2 NPU, see npu.nix) that previously had zero CPU/PCIe power management, unlike latitude-7212's tlp.nix. Enables TLP (powersave governor + balance_performance/power EPP, turbo boost off on battery, AMD PMF platform_profile set to low-power on battery where the firmware exposes it, USB autosuspend, audio power-save) and disables power-profiles-daemon — mostly insurance, since tux runs Hyprland (desktop.nix), which never sets ppd's mkDefault true the way gnome.nix does for latitude-7212. Wifi power_save is forced off on both AC and battery to match the existing suspend/resume workaround in common.nix rather than fight it. Unlike latitude, tux hasn't fought any PCIe/NVMe stability bugs, so PCIE_ASPM_ON_BAT = "powersave" is enabled (AC left at platform default) — if wifi or NVMe gets flaky on battery after this, that's the first setting to drop.

TUXEDO vendor driver (tux)

modules/nixos/tuxedo.nix — tux is a TUXEDO InfinityBook Pro 14 AMD Gen10. Enables hardware.tuxedo-drivers (Fn-lock, keyboard backlight, the vendor kernel module TUXEDO Control Center talks to) mainly for settings.charging-profile = "stationary", which caps battery charging at 80% — drop to "balanced" (90%) if that's too conservative. The benign ACPI BIOS errors seen at boot (NPCF.ACBT/ATPP/DBAC, _SB.ACDC.RTAC, _REG.DPMF) are Fn-key ACPI-query noise confirmed harmless by TUXEDO's own FAQ and unrelated to this driver — not worth chasing. hosts/tux/default.nix also carries two kernel params reported for the Gen9 AMD sibling board (unverified on this exact Gen10 board, revert if they don't help or cause regressions): acpi.ec_no_wakeup=1 (spurious EC wakeups from suspend) and amdgpu.dcdebugmask=0x10 (amdgpu Wayland performance workaround). Separately, kernel 7.1.2–7.1.3 had a known suspend-breaking bug on this exact board (Motorcomm YT8531S ethernet driver interrupt handling); fixed upstream and merged by 7.2, which is what npu.nix's linuxPackages_latest currently resolves to — no action needed unless a future pin lands on an affected version.

VM/container storage on /home (tux)

hosts/tux/default.nix redirects Podman's (docker-compat, common.nix) and libvirtd's (virt-manager) bulk storage off cryptroot (196G root partition) onto /home. Not /datengrab (hardware-configuration.nix): that drive is for data meant to survive a host disk wipe, and VM/container storage is disposable, host-tied working state — the wrong fit despite datengrab having the most free space. Prompted by a 2026-09-04 incident where cryptroot filled to 0 free mid-rebuild; nix-collect-garbage -d alone reclaimed 111G, but VM disks and container images growing unbounded on a 196G root was the underlying risk.

  • Podman: virtualisation.containers.storage.settings.storage.graphroot = "/home/containers/storage" — a plain storage.conf setting, no filesystem tricks needed.
  • libvirtd: fileSystems."/var/lib/libvirt/images" bind-mounts to /home/libvirt/images (directory created via systemd.tmpfiles.rules) — VM XML definitions keep referencing /var/lib/libvirt/images/*.qcow2 unchanged, since the bind mount keeps that path but backs it with /home.

Migrating existing data is a manual, one-time step (not automated by the Nix config): stop podman/libvirtd (shut down VMs first), rsync -aHAX the old /var/lib/containers/storage and /var/lib/libvirt/images trees to their /home targets, verify, delete the old copies on cryptroot, then nixos-rebuild switch to activate the new graphroot/bind-mount — doing the rebuild before migrating means Podman/libvirtd start pointing at empty new locations while the old data sits stranded on root.

Hibernate / swap (tux)

Added 2026-09-18 after discovering tux had no swap at all (hardware-configuration.nix shipped swapDevices = [ ]) — hibernate wasn't just untested, it was categorically impossible without swap to hold the memory image. hosts/tux/default.nix now declares a 64 GiB swapfile (matching this machine's 64 GiB RAM) at /home/swapfile — not / (cryptroot, only 199.5G and already prone to filling up, see the VM/container storage incident above) — and boot.resumeDevice = "/dev/mapper/crypthome". This still works for resume even though the swapfile isn't on root: crypthome is already unlocked during the initrd (it's in boot.initrd.luks.devices, needed just to mount /home), so it's available at the same point in boot that root would be, and hibernate's kernel-level resume restores the entire prior running state (root included) directly from the image — it doesn't need root mounted normally first.

boot.kernelParams also carries resume_offset=419014656 — the swapfile's physical location within crypthome, needed for the kernel to find the hibernation image at boot. Computed after the swapfile existed (chicken-and-egg: hms once to create the file, sudo filefrag -v /home/swapfile | awk '$1=="0:"{print substr($4,1,length($4)-2)}' to get the offset, then set it and hms again). ext4 doesn't relocate file blocks, so this stays valid indefinitely — only needs recomputing if the swapfile is ever deleted and recreated.

After the reboot needed to pick up resume=/resume_offset=, systemctl hibernate still failed with Call to Hibernate failed: No such file or directory — a second, separate bug. systemd-logind (v261) sandboxes itself with ProtectHome=, hiding /home (and the swapfile in it) from its own mount namespace; logind needs to stat/open the swapfile itself to validate the hibernation location, so with /home masked out that lookup fails with ENOENT — confirmed via the read-only busctl … CanHibernate query failing identically, with no corresponding entry in the logind journal. Stopgap fix: systemd.services.systemd-logind.serviceConfig.ProtectHome = lib.mkForce false; in hosts/tux/default.nix.

This is a stopgap, not the end state — swap living under /home will keep colliding with systemd's hardening as it tightens further. The clean fix is a dedicated swap partition (no resume_offset needed, and the device lives under /dev/mapper/, which ProtectHome doesn't touch), which needs live disk surgery (shrink crypthome, repartition, new LUKS volume) — not yet scheduled, see docs/wiki/tux-hibernate-setup.md for the plan and the open question of which disk to shrink from.

Hibernate is still not fully verified end-to-end as of this writing — the ProtectHome fix is written but not yet rebuilt/retested on the machine.

The power menu (waybar's custom/powermenu button → wlogout, or quickshell's PowerMenu.qml — see Desktop bar) includes a Hibernate action (systemctl hibernate) that depends on this setup being complete.

Gaming desktop (big-rig)

Converted 2026-08-30 from CachyOS/Arch (home-manager-only) to a full NixOS install — hosts/big-rig/ home.nix kept its shape from the Arch days; only the bits NixOS now manages declaratively changed (logiops, Brave's managed policy, Steam — all just work via common.nix/desktop.nix like every other desktop NixOS host — Steam/Bluetooth/libvirtd live in desktop.nix, not common.nix, so the headless homelab-n cluster doesn't carry them). VMs are libvirt/QEMU via virt-manager (desktop.nix, same as every other desktop host), not VirtualBox. No known hardware stability quirks (unlike latitude-7212), so it imports only common.nix + desktop.nix + kanata.nix + logiops.nix + nvidia.nix + streamdeck.nix — none of the tablet's iwlwifi/NVMe-APST/thermal/PSR workaround modules.

  • NVIDIA GPU — custom.nvidia = true (modules/nixos/nvidia.nix); the Hyprland NVIDIA env vars (LIBVA_DRIVER_NAME etc.) are set via custom.hyprlandExtraLua in hosts/big-rig/home.nix, as they were on Arch.
  • Kanata without homerow mods — custom.kanataConfig = ../../configs/kanata-simple.kbd (tux uses the default configs/kanata.kbd with GACS mods instead); a gaming desktop makes accidental homerow-mod activation more disruptive than convenient.
  • Elgato Stream Deck — modules/nixos/streamdeck.nix installs StreamController, its udev rules (user access to the deck's hidraw device; replug the deck after the first switch), and a streamcontroller -b systemd user service on graphical-session.target. Buttons/pages/plugins are configured in the app and stored in ~/.var/app/com.core447.StreamController — not declarative. Also imported on tux; import it on any other NixOS desktop a deck is plugged into. On macOS use Elgato's official app instead (brew install --cask elgato-stream-deck). CachyOS bridge: while big-rig still runs standalone home-manager, hosts/big-rig/home.nix installs the package + the same user service instead, but HM can't install udev rules — copy them once by hand (fish): sudo cp (find (dirname (dirname (readlink -f (command -v streamcontroller)))) -name '*.rules') /etc/udev/rules.d/; and sudo udevadm control --reload; and sudo udevadm trigger, then replug the deck. Remove that file again once big-rig is on NixOS.
  • Login password — users.users.jz.initialHashedPassword in hosts/big-rig/default.nix ships as an obvious placeholder string; replace it with a real mkpasswd -m sha-512 hash before building the unattended installer for this host (the interactive bootstrap installer prompts for a password itself, so this only matters for the unattended path).

Headless k3s cluster (homelab-1/homelab-2/homelab-3)

Three identical NUC-style boxes, no display/keyboard ever attached. homelab-1 runs services.k3s.role = "server" (single-node sqlite datastore, not the HA embedded-etcd mode); homelab-2/homelab-3 run role = "agent" against it. They import common.nix + modules/nixos/server.nix instead of desktop.nix — no display manager, audio, or Bluetooth, since nothing pulls them in. Provisioned entirely via the homelab-n unattended installer; there is no interactive install path for these (no console to use one at).

  • Access is SSH-key-only by default — modules/nixos/server.nix sets PasswordAuthentication = lib.mkDefault false and no login password is installed for jz (no initialHashedPassword, unlike big-rig/latitude-7212). Replace the placeholder key in that module with a real one before building any installer ISO that uses it — on a fresh install it's the only way in. homelab-1 opts in to password auth as well (hosts/homelab-1/default.nix); its password is set imperatively once over the key (sudo passwd jz), since a declarative hash wouldn't apply to an already-existing user with mutableUsers on.
  • Claude Code has its own key per host (claude@homelab-1, claude@homelab-2, declared in each host's default.nix on top of server.nix's key). The private keys are passphrase-less and live in ~/Projects/homelab-{1,2}_ed25519 on the workstation. With jz's NOPASSWD sudo that's root on the box, so remove the host's line to revoke it.
  • Passwordless sudo for jz, scoped to these hosts only (security.sudo.extraRules in server.nix) — there's no console to type a sudo password at, and it's what lets nixos-rebuild --target-host apply changes non-interactively. Justified because access is gated by the SSH key above — on homelab-1, which also accepts a password, that password is effectively root's.
  • Lingering for jz (users.users.jz.linger in server.nix) — keeps /run/user/1000 alive after the last SSH session ends. home-manager's tmux secureSocket puts the socket there, so without it a dropped connection deletes the socket and a detached tmux session becomes unreachable from the next login (recover one with tmux ls; pkill -USR1 -x tmux).
  • Wi-Fi is declared, not interactive — unlike tux/latitude-7212 (configured once with nmtui), a headless box needs to reconnect after every reboot with nobody there to help it. server.nix declares NetworkManager profiles for both chaos-5g and chaos-2g, with the PSKs pulled at activation from /etc/nixos-secrets/wifi.env — a plain root-only file living outside the Nix store and outside git, written directly onto the disk by the installer (see below). Same reasoning for services.k3s.tokenFile → /etc/k3s-token, the shared cluster join-secret.
  • Why not sops-nix for any of the above: sops needs each host's SSH-host-key-derived age public key added to .sops.yaml before anything can be encrypted for it, and that key only exists once the machine has been installed once. For a brand-new host that's a chicken-and-egg problem sops-nix doesn't solve on its own — tux's own Pangolin secret (below) was added after tux already existed and had a real host key. Migrating these to sops later (once the hosts exist) is straightforward if tighter secret hygiene is ever wanted.
  • Pangolin, in site mode (unlike tux, which runs client) — these nodes host services on the k3s cluster, so they expose them outward through Pangolin rather than just reaching other sites. Each node has its own site ID (not secret, committed in its default.nix) and its own site secret. modules/nixos/pangolin.nix gained a customPangolin.secretFile option for exactly this case: an alternative to its normal sops-nix path, pointed at /etc/nixos-secrets/pangolin.env — same install-time-written-file mechanism as the k3s token/Wi-Fi PSKs above, for the same chicken-and-egg reason.
  • Deploy from another machine, no login needed — just deploy homelab-1 runs nixos-rebuild switch --target-host jz@homelab-1.local --elevate=sudo (just reboot=true deploy homelab-1 runs nixos-rebuild boot instead, reboots the node and waits for SSH, for kernel updates): it builds locally (these NUCs are weak) and pushes + activates over SSH using the same key and passwordless sudo above. The push needs jz in the target's nix.settings.trusted-users (server.nix), since the daemon otherwise rejects paths built elsewhere as unsigned — a node installed before that setting existed must be updated once by building on the node itself (see docs/wiki/homelab-cluster-install.md). server.nix also enables avahi so <hostname>.local resolves on the LAN without static IPs or /etc/hosts entries.
  • Pull-based deploys (comin, on homelab-1, homelab-2, hetzner-robot-1) — modules/nixos/comin.nix runs comin: every 5 minutes the host fetches tools/nixconfigs over HTTPS, builds its own nixosConfigurations entry and switches to it. It follows the deploy branch, which deploy-gate.yml fast-forwards to a main commit only after all four servers build, so nothing needs SSH into the host and CI holds no host credentials. Pushing to testing-<host> tries a commit with switch-to-configuration test (not the boot default). comin doesn't reboot; kernels still need just reboot=true deploy. Token: a read-only Forgejo access token (read:repository) in /etc/nixos-secrets/comin-token (root, 0600). Its exporter is scraped as job comin (localhost:4243). This covers every provisioned server; homelab-3's config exists and is built by the gate, but the machine isn't deployed yet. A host is bootstrapped onto comin with one manual just deploy <host> after the token file is in place; from then on it self-updates.
  • k3s ports — homelab-1 opens 6443/tcp (API server) and 10250/tcp (kubelet) inbound; the agents open only 10250/tcp. All three open 8472/udp (flannel VXLAN, pod-to-pod traffic). kubectl from another machine on the LAN talks to https://homelab-1.local:6443. tux also carries the flux CLI (fluxcd, in hosts/tux/home.nix) for GitOps bootstrap/reconcile.
  • RustFS object store on homelab-2 (modules/nixos/rustfs.nix → customRustfs, wrapping the nixpkgs services.rustfs module) — S3-compatible backup target, data in /var/lib/rustfs on the 1 TB internal SSD. Listens on loopback only (127.0.0.1:9000 S3, :9001 console — ssh -L 9001:127.0.0.1:9001 jz@homelab-2.local); no firewall port. Remote clients reach it through a Pangolin private Host resource (destination 127.0.0.1:9000, alias s3.homelab-2.internal, served by homelab-2's own site tunnel). Admin credentials: /etc/nixos-secrets/rustfs.env (root-only, written once by hand, not in git). Adding a backup client is one line in customRustfs.backupClients: the rustfs-provision oneshot then creates a bucket, a policy scoped to it, and a user with that policy, all named after the entry, and generates the user's keys into /etc/nixos-secrets/rustfs-<name>.env (kept across runs). Hand them over with e.g. ssh jz@homelab-2.local sudo cat /etc/nixos-secrets/rustfs-<name>.env | kubectl create secret generic … --from-env-file=/dev/stdin. Removing an entry leaves bucket/user/data in place on purpose. Provisioning uses mc admin (RustFS speaks the MinIO admin API for users/policies; mc admin info isn't supported). nixpkgs marks minio-client insecure since 2026-10 (abandoned upstream), so rustfs.nix permits that one exact version; it only talks to 127.0.0.1. Replace it when a maintained client for the admin API exists.

Remote server (hetzner-robot-1)

Hetzner Robot dedicated server (i7-8700, 128 GB RAM, 2× 1 TB NVMe), 65.21.79.239, the planned replacement for the datengrab VPS. Public internet, so it differs from the homelab boxes:

  • Install: no console or USB, so neither installer ISO applies. just install-remote hetzner-robot-1 65.21.79.239 (as root on the target via your SSH key) runs nixos-anywhere: it SSHes in as root to whatever Linux is running (the installimage Ubuntu, or Hetzner's rescue system), kexecs into a NixOS installer, partitions with disko, installs and reboots. It wipes both disks. If the box doesn't come back, boot the rescue system from Hetzner Robot and check the network config.
  • Disks: hosts/hetzner-robot-1/disko.nix. mdadm RAID 1 over both NVMe disks (by-id paths), one ext4 /. The board boots legacy BIOS: each disk has a 1 MiB BIOS boot partition, disko adds both to boot.loader.grub.devices, so GRUB is on both mirrors. 8 GiB swapfile as an OOM buffer.
  • Kernel: linuxPackages_latest, required. nixos-anywhere's installer (newer kernel + mdadm) writes the logical block size into an md superblock field that older kernels still treat as padding; 6.18's initrd rejected the array ("does not have a valid v1.2 superblock") and the first boot hung without a root filesystem. Hetzner's rescue system (kernel 6.12) can't assemble the array either: read a mirror directly instead, read-only, e.g. losetup -r -f --show -o $((264192*512)) /dev/nvme0n1p2 and mount -o ro,noload /dev/loopN /mnt (264192 = the md data offset in sectors, mdadm --examine shows it).
  • Network: systemd-networkd instead of NetworkManager. Hetzner's routed setup: /32 with the gateway on-link, IPv6 /64 via fe80::1, Hetzner's resolvers, matched by MAC.
  • Metrics + logs: from the first boot (standard for servers, see Monitoring). The two password files live in ~/Projects/hetzner-robot-1-secrets/etc/nixos-secrets/ and the install recipe copies them along (--extra-files, chowned to root).
  • Access: server.nix as for the homelab (key-only SSH, no root login, NOPASSWD sudo for jz), plus a per-host Claude key. Avahi is forced off (it would publish mDNS on the public IP). Pangolin site (customPangolin, mode site, site ID ltotf6ii8oyaten): the box hosts workloads exposed through Pangolin; outbound tunnel only. Secret line in /etc/nixos-secrets/pangolin.env (root 0600, copy in hetzner-robot-1-secrets), as on homelab-2. Firewall: SSH plus 80/443 (Traefik); services open further ports as they move over. Admin SSH is on port 2342 (services.openssh.ports): 22 is code.zeidler.dev's git SSH. ~/.ssh/config needs Port 2342 for the host alias and the addresses (just deploy hetzner-robot-1 65.21.79.239); the Hetzner rescue system (nixos-anywhere reinstall) still listens on 22.
  • CI runner: ci-runner.nix runs the Forgejo Actions runner for code.zeidler.dev (hetzner-robot-1-runner-1, same labels as datengrab-runner-1, capacity 4) inside a KVM microVM (microvm.nix, 8 vCPU / 24 GiB, 250 GiB sparse /var/lib image). Jobs run privileged DinD, so the CI's Docker stays inside the VM (the host's own Docker, below, never sees a job). tap vm-ci 10.200.0.1/24 → VM 10.200.0.2, NAT via eno1. The guest journal is forwarded to its console, i.e. journalctl -u microvm@ci-runner on the host (and Loki). The runner is created in Forgejo (Site Administration → Actions → Runners) and declared as a connection in the runner config (UUID in ci-runner.nix, no register step); its token goes alone into /var/lib/microvms/ci-runner/secrets/runner-token (microvm-owned, 0600; qemu serves it read-only into the VM, systemd loads it as a credential), then systemctl restart microvm@ci-runner.
  • Deploy: just deploy hetzner-robot-1 65.21.79.239 (SSH on 2342, see Access) (the recipe takes an address when <host>.local doesn't resolve).
  • Compose stacks (services moved from datengrab): modules/nixos/compose-stacks.nix turns customComposeStacks.stacks.<name> = { directory; composeFiles; extraComposeFiles; } into compose-<name>.service, a oneshot running docker compose up -d --wait after docker.service (retried on failure, stop on stop). The directory is on-disk state holding the upstream compose file and its secrets (root-only); git holds only override files (extraComposeFiles, a change re-runs up). The unit is skipped until the directory's first compose file exists. Enabling the module swaps common.nix's podman for real Docker (journald as default log driver, weekly prune that never touches volumes). ip-forward-no-drop in ci-runner.nix keeps Docker from setting iptables' FORWARD policy to DROP, which would cut the CI VM's NAT.
  • Ingress: ingress.nix, native Traefik with the Docker provider (exposedByDefault = false, socket access via the docker group). Entrypoint https and resolver letsencrypt match datengrab's names, so stacks keep their labels. HTTP-01 on :80: a hostname gets its certificate only once DNS points here; before that, test with curl -k --resolve <host>:443:65.21.79.239.
  • Harbor (harbor.nix, registry.zeidler.dev): upstream offline installer in /srv/harbor (harbor.yml with secrets, 0600 → ./prepare → docker-compose.yml + common/config), data volume /srv/harbor/data, image blobs in the Hetzner Object Storage bucket chaosharbor. stacks/harbor/docker-compose.override.yml switches all containers to journald (no more harbor-log boot race, logs reach Loki), drops nginx's published port and adds the Traefik labels. Config change/upgrade: edit harbor.yml, systemctl stop compose-harbor, ./prepare, systemctl start compose-harbor.
  • code.zeidler.dev (forgejo-zeidler.nix): Forgejo 16.0.6 + MySQL 8.4.11, pinned by digest. The compose file is in git (stacks/forgejo-zeidler), run with --project-directory /srv/forgejo-zeidler (composeFiles = [ ], requiredFile .env): .env holds the DB user/password, data/{git,gitea,ssh} the repos/LFS, app.ini + attachments/packages and the SSH host keys, mysql/ the datadir. Git SSH on 22 (container OpenSSH), web on 3001 behind Traefik.
  • Stack backups: modules/nixos/stack-backups.nix, customStackBackups.jobs.<name> = { preBackup; paths; exclude; } → stack-backup-<name>.service + nightly timer (03:30 + up to 15 min). preBackup writes dumps into $STAGE; then restic and the Pangolin machine client run together in a dockerTools image (the client's DNS override and routes stay in the container's network namespace), to s3:http://s3.homelab-2.internal:9000/<name>-backup/<name> on homelab-2's RustFS (bucket from customRustfs.backupClients there). Retention 7 daily / 4 weekly / 6 monthly, restic unlock before forget, restic check every run. Success → stack_backup_last_success_timestamp_seconds{backup} via node-exporter's textfile collector (StackBackupStale in chaos-cloud). Secrets: /etc/nixos-secrets/pangolin-backups.env (machine client) and restic-<name>.env (RESTIC_PASSWORD + the RustFS keys); a job is skipped until its env file exists. Harbor backs up a pg_dump -Fc of registry (checked with pg_restore --list), harbor.yml, data/secret and data/ca_download. code.zeidler.dev (04:00): a mysqldump --single-transaction (checked for -- Dump completed), .env and data/ without indexers/queues/logs, bucket forgejo-zeidler-backup (repo /forgejo-zeidler; datengrab-backup's /forgejo there is the old history).

Custom installer ISO (installer)

Instead of the manual steps above, build a reusable installer ISO with this flake and a guided bootstrap script baked in. Boot it on any UEFI machine and run one command.

Build the image (on any machine with Nix + flakes):

nix build .#installer-iso

(short alias for .#nixosConfigurations.installer.config.system.build.isoImage.) This is an x86_64 Linux build. The ISO lands at result/iso/nixconfig-installer.iso.

Write it to a USB stick (Linux). This erases the stick — identify the device carefully; dd to the wrong disk is irreversible.

# 1. Find the stick by size/model (USB sticks show TRAN=usb).
#    Use the whole-disk name (/dev/sdX), NOT a partition (/dev/sdX1).
lsblk -dpno NAME,SIZE,MODEL,TRAN

# 2. Unmount anything auto-mounted from it, and clear stale signatures.
sudo umount /dev/sdX?*  2>/dev/null || true
sudo wipefs -a /dev/sdX

# 3. Write and flush. (Do NOT use oflag=direct — it can drop the final partial
#    block and produce a truncated ISO that boots to a bare `grub>` prompt.)
sudo dd if=result/iso/nixconfig-installer.iso of=/dev/sdX bs=4M status=progress conv=fsync
sync

# 4. Verify the stick matches the ISO byte-for-byte.
sudo cmp -n "$(stat -c%s result/iso/nixconfig-installer.iso)" \
           result/iso/nixconfig-installer.iso /dev/sdX && echo "OK: stick matches ISO"

Boot the stick, then get online first — nixos-install downloads nixpkgs and all packages, so the installer needs working internet before you run bootstrap:

# wired: automatic via DHCP — just plug in
# wifi:  the ISO ships NetworkManager + redistributable firmware (incl. Intel iwlwifi)
nmcli radio wifi on
nmtui                                              # TUI, or:
nmcli device wifi connect "<SSID>" password "<PW>"
ping -c1 nixos.org                                 # confirm connectivity

The ISO enables hardware.enableRedistributableFirmware (see installer/iso.nix) so WiFi adapters like Intel iwlwifi work in the live installer — the stock minimal ISO ships less firmware and leaves such cards unavailable. (enableAllFirmware is intentionally avoided: it pulls fragile blobs such as facetimehd-calibration that fail to build.) If nmcli device still shows the adapter unavailable, check rfkill list (run rfkill unblock wifi) or fall back to wired.

Then run the embedded bootstrap:

sudo bootstrap [DISK] [HOST]

# examples
sudo bootstrap                    # prompts for disk + hostname, installs nixos-template
sudo bootstrap /dev/nvme0n1       # installs the template onto that disk
sudo bootstrap /dev/sda nixos-vm  # different disk + a specific host

bootstrap (defined in installer/bootstrap.sh) first verifies it can reach cache.nixos.org (refusing to touch any disk if you're offline — override with SKIP_NET_CHECK=1), confirms before wiping, partitions the disk UEFI/GPT into ESP + LUKS root (200 GiB) + LUKS /home (rest) — a separate encrypted /home so you can reinstall the OS later without losing it. Root and /home get the same passphrase and the systemd initrd unlocks both with one prompt; nixos-generate-config captures both boot.initrd.luks.devices entries and the /home mount. Tunables: ENCRYPT=0 (no encryption), SEPARATE_HOME=0 (put /home on /), ROOT_END=<size> (root size, default 200GiB).

Reinstalls keep your data: on a disk that already has a /home partition, bootstrap detects it and asks whether to keep /home or reformat the whole disk (or force it with KEEP_HOME=1). Keeping it reformats only root + ESP and re-uses the existing encrypted /home untouched. It then copies the embedded flake to /mnt/etc/nixconfig, drops the generated hardware config into hosts/<HOST>/, and runs nixos-install. It also asks for this machine's hostname and drops a ~/finish-setup.sh (hostname baked in) that graduates the template into a real hosts/<hostname>/ on first boot (see the template model). After install it prompts to set the login user's password, generate an ed25519 SSH key for them, and reboot (all optional; the user defaults to jz, override with the USERNAME env var). On an encrypted machine you'll be prompted for the LUKS passphrase at every boot, before systemd-boot hands off to the kernel. The flake is shipped read-only inside the ISO at /etc/nixconfig (see installer/iso.nix), so the installer works fully offline for evaluation (package downloads still need network).

Both tux and latitude-7212 (hosts/<name>/default.nix) enable boot.plymouth.enable (default spinner theme) plus quiet + consoleLogLevel = 3, so that bare LUKS prompt becomes a themed graphical splash with masked password entry. Relies on boot.initrd.systemd.enable from common.nix for clean systemd-initrd integration; Esc at boot still drops to verbose text if a boot ever needs debugging. Not applied to nixos-template/nixos-vm — the installer flow benefits from seeing raw boot text.

The ISO is reusable across machines and hosts — HOST defaults to nixos-template but accepts any host under hosts/. Because the ISO embeds a snapshot of the flake, rebuild it after changing your config to pick up the changes.

After install, log in as jz, apply the home-manager profile, and commit the generated hardware config back to the repo (same final two steps as the manual install above).


Fully unattended installer ISO (installer-unattended)

A zero-interaction variant for imaging a machine (currently hard-wired to install host latitude-7212). Boot it and walk away: it auto-connects a baked-in wifi, waits for internet, requires exactly one internal disk to be present (removable media — the USB stick — is excluded; more than one internal disk and it stops rather than guess), gives a 20-second keypress-to-abort window on tty1, then wipes and installs. No encryption, so the only baked secret is the wifi PSK (plus an optional login-password hash).

⚠️ The stick becomes a secret. The wifi PSK (and any login hash) are baked into the image in plaintext. Anyone with the stick can read them. Keep them out of git via the git-ignored secrets file below, and treat the physical stick accordingly.

1. Create the secrets file (git-ignored — see .gitignore; may live anywhere):

cp installer/unattended-secrets.example.nix installer/unattended-secrets.nix
$EDITOR installer/unattended-secrets.nix        # wifiSSID, wifiPSK, optional userPasswordHash
# hash for sudo (optional):  nix run nixpkgs#mkpasswd -- -m sha-512

The secrets are read impurely from $NIXCFG_UNATTENDED_SECRETS — never from the flake source — so they stay out of git entirely. In pure eval (nix flake check, the security-scan CI) that env var is empty, so the .#installer-unattended-iso output simply doesn't exist and nothing else is affected.

2. Build and flash (same dd procedure as the interactive ISO above). The build needs --impure and the env var pointing at your secrets file — the justfile wraps that:

just installer-unattended        # wraps the --impure + env-var build; guards a missing secrets file
# or point at a secrets file elsewhere:
just secrets=/run/keys/wifi.nix installer-unattended

# equivalent raw command:
NIXCFG_UNATTENDED_SECRETS="$PWD/installer/unattended-secrets.nix" \
  nix build --impure .#installer-unattended-iso
# result/iso/nixconfig-installer-unattended.iso  → dd to the stick

3. Boot the target. It runs installer/unattended-install.sh as unattended-install.service on tty1: connectivity check → largest-internal-disk pick → 20 s abort window → GPT (ESP + ext4 root) → nixos-generate-config → embedded flake copied to /mnt/etc/nixconfig with the generated hardware config injected into hosts/latitude-7212/ → nixos-install --no-root-passwd → optional chpasswd from the baked hash → reboot. A /etc/unattended/done marker + a service ConditionPathExists guard stop it re-running.

Safety notes: the disk picker filters on RM==0 and TRAN!=usb so it can only target non-removable, non-USB disks (never the boot stick — some sticks report RM=0, hence the transport check); if nothing qualifies it aborts rather than wiping anything. Before wiping, it releases the target disk (swapoff, unmount, and vgchange -an on any LVM volume group found on it — e.g. a preinstalled Ubuntu's ubuntu-vg, which the live ISO auto-activates and which otherwise makes wipefs fail with "Device or resource busy"). To retarget a different host or partition scheme, edit installer/unattended.nix (service environment) and installer/unattended-install.sh. Since it's baked from a flake snapshot, rebuild the ISO after changing your config. Test it in a VM before trusting it on real hardware.

Debugging a stall/freeze. The install script is instrumented so a hang leaves evidence instead of a blank screen:

  • A diagnostics snapshot (diag_dump) prints at start and on failure — RAM/swap, disks, network + wifi driver, recent iwlwifi/MCE dmesg lines, CPU temps, and any prior-boot crash log from pstore.
  • All output is mirrored to a logfile, and once the target disk is mounted a background sampler writes a state snapshot (RAM, temps, last iwlwifi/MCE/OOM line, store size) every 5 s to the target disk (sync'd), so a hard freeze mid-download leaves a breadcrumb trail. Everything lands in /var/log/unattended/ (install.log, samples.log, dmesg.txt) — which survives a reboot and persists onto the installed system. After a freeze, reboot the live stick, mount the root partition, and read those files.
  • Mirror logs onto the boot USB stick (easiest to retrieve). The ISO filesystem is read-only, so add a spare FAT32 partition labelled NIXLOG to the stick's free space once (after writing the ISO). When present, the installer mounts it (-o sync) and mirrors install.log + samples.log (+ dmesg.txt on failure) into /unattended/ on it — from the very first line, independent of the target disk, so even an early "no internet" abort or a download-phase freeze is captured. Then just pull the stick, plug it into any machine, and open the plain-text logs (no need to boot the target or pull its internal disk). The easiest path is just flash [disk] (default /dev/sda), which builds the unattended ISO, dds it to the stick, and adds the NIXLOG partition in one step (it refuses any non-removable disk and makes you confirm). To prep an already-flashed stick by hand instead (use sfdisk, not sgdisk — gptfdisk can't parse the ISO's isohybrid GPT and dies with "Invalid partition data!"):
    # /dev/sdX is the boot stick. The ISO occupies partition(s) at the front; grow
    # the GPT to fill the stick, append ONE partition in the freed space, FAT32 it.
    sudo sfdisk --relocate gpt-bak-std /dev/sdX   # move backup GPT to the true disk end
    printf ',,L\n' | sudo sfdisk --append /dev/sdX  # append a partition spanning the free space
    sudo mkfs.vfat -F 32 -n NIXLOG /dev/sdXN      # N = the new partition number
    
    (Label matching is set by USB_LOG_LABEL in unattended.nix's service env. No stick partition → logging is unchanged. The stick already holds the baked wifi PSK, so it's a secret either way; the password hash is kept out of the log.)
  • nixos-install runs verbose with --option stalled-download-timeout 90 --option connect-timeout 15, so a stalled fetch fails loudly (hitting the error trap + debug shell) instead of hanging forever.
  • DEBUG=1 (default in unattended.nix's service env) adds a timestamped per-command trace; set it to "0" to quiet things once the machine installs cleanly. boot.consoleLogLevel = 7 surfaces kernel messages (firmware crashes, MCE, OOM) on the console.
  • Swap (the tablet is low-RAM). The live installer ships no swap, and the download/unpack phase ran this tablet out of memory (~130 MiB free, thrashing to a crawl — which looked like a hardware freeze for a long time: downloads stall, new gettys won't spawn, the box is unresponsive but the kernel is still alive). Fixed two ways: zramSwap.enable (compressed in-RAM swap, from early boot, both ISOs) and a real swapfile the install script puts on the mounted target (SWAP_MB, set to 16384 in unattended.nix since the config compiles several non-cached Rust/Go packages that peak multi-GB on the 8 GB tablet; script fallback 8192; removed before reboot). The installed host uses zramSwap at runtime too (GNOME on low RAM). If a future install still starves, raise SWAP_MB further.

Intel 8265 wifi (modules/nixos/iwlwifi-stable.nix). The Latitude's Intel 8265 is served only by iwlwifi (no alternative driver exists to switch to), and its firmware crashes / stalls downloads under power management and Tx aggregation — the root cause behind the "stuck downloading" freezes. The module has two profiles via stabilityWorkarounds.iwlwifi.profile. "aggressive" (default, used by both installer ISOs) throws everything at the firmware to guarantee a one-shot install completes: power_save=0, bt_coex_active=0, swcrypto=1, 11n_disable=8, uapsd_disable=1, power_scheme=1 plus a global pcie_aspm=off. "relaxed" (the latitude-7212 host) keeps only the cheap power-save options (power_save=0, uapsd_disable=1, power_scheme=1) and restores the throughput / battery / Bluetooth-coexistence features the install-time hammers gave up — BT coexistence matters because this tablet's only keyboard is Bluetooth, and dropping the global pcie_aspm=off reclaims idle battery (the NVMe freeze is APST, not PCIe ASPM, so it isn't reintroduced). The host can roll back if the relaxed profile stalls. If the installer still stalls on "aggressive", escalate 11n_disable=8 → 11n_disable=1 (disables 802.11n entirely: slower, most stable). Only add iwlwifi params that modinfo -p iwlwifi lists — an unknown option makes the module fail to load and wifi vanish.

big-rig variant (installer-unattended-big-rig)

A second, separate unattended ISO targets host big-rig instead of latitude-7212 — installer/unattended-big-rig.nix, a sibling of installer/unattended.nix rather than a parameterized single module (same reasoning as tlp.nix vs tlp-tux.nix: the two hosts need different hardware workarounds). big-rig is a normal desktop with none of the Latitude tablet's known freeze bugs, so none of iwlwifi-stable.nix / nvme-stable.nix / thermald.nix / display-stable.nix are imported; zramSwap is kept as a cheap, harmless safety net. Build isn't throttled (CORES=0 MAX_JOBS=4 NO_TURBO=0) since there's no known power/thermal constraint to work around, unlike the tablet.

just installer-unattended-big-rig   # or: just secrets=/path/to/secrets.nix installer-unattended-big-rig
just flash-big-rig                  # build + write to /dev/sda + add the NIXLOG log partition
# equivalent raw command:
NIXCFG_UNATTENDED_SECRETS="$PWD/installer/unattended-secrets.nix" \
  nix build --impure .#installer-unattended-big-rig-iso

Same shared installer/unattended-secrets.nix (wifi SSID/PSK), same log-mirroring/diagnostics instrumentation (installer/unattended-install.sh is entirely shared — it's already $TARGET_HOST-parameterized, not Latitude-specific).

⚠️ Desktop disk warning. The installer requires exactly one internal (non-removable) disk to be present — fine on a single-drive laptop, but on a multi-drive desktop (e.g. a Windows drive and/or a data disk alongside the intended target) it refuses to guess and drops to a shell instead. Physically disconnect every drive except the intended target before booting the stick.

⚠️ Set a real login password first. hosts/big-rig/default.nix's users.users.jz.initialHashedPassword ships as an obvious placeholder — generate a real one (nix run nixpkgs#mkpasswd -- -m sha-512) and commit it before building this ISO, or the install will finish with no usable login (see Gaming desktop (big-rig)).

homelab-n variant (installer-unattended-homelab)

A third variant, for the headless k3s cluster. Unlike installer-unattended/installer-unattended-big-rig (one module per host, forked because each targets different hardware), installer/unattended-homelab.nix is one module reused for all three identical NUCs — the only thing that differs between them is which host gets installed, so that's read impurely too (NIXCFG_UNATTENDED_HOST), the same trick already used for the secrets path. It also connects to either chaos-5g or chaos-2g (whichever is in range), and — unlike the other two variants — bakes secrets the installed system needs going forward, not just the installer itself: the k3s cluster join-token and the Wi-Fi PSKs, written by installer/unattended-install.sh straight onto the target disk as plain root-only files (never into the Nix store, never into git — see the cluster section above for why sops-nix isn't used here).

just installer-unattended-homelab homelab-1   # build the ISO for one specific node
just flash-homelab homelab-1                  # build + write to /dev/sda + add the NIXLOG log partition
just flash-homelab homelab-2 /dev/sdb         # different node, different stick
# equivalent raw command:
NIXCFG_UNATTENDED_SECRETS="$PWD/installer/unattended-secrets.nix" \
NIXCFG_UNATTENDED_HOST=homelab-1 \
  nix build --impure .#installer-unattended-homelab-iso

Extends the shared installer/unattended-secrets.nix with wifi5gPsk/wifi2gPsk (same value twice if both bands share a password), k3sToken (openssl rand -hex 32), and pangolinSecrets (one site secret per node, keyed by hostname — see installer/unattended-secrets.example.nix). Same log-mirroring/diagnostics instrumentation as the other two (installer/unattended-install.sh is entirely shared).

⚠️ Set a real SSH key first. modules/nixos/server.nix's users.users.jz.openssh.authorizedKeys.keys ships with an obvious placeholder — there's no password and no console on these boxes, so an unreplaced placeholder means no way in at all after install. Replace it and commit before building this ISO.

⚠️ Rebuild for each node. Because the flake is baked into the ISO as a snapshot, build a separate ISO per NIXCFG_UNATTENDED_HOST (homelab-1, then homelab-2, then homelab-3) — don't reuse one stick's image across all three, or they'll all install as the same host.

Watching it happen, remotely. The other unattended ISOs assume you're standing in front of the machine and only leave a mirrored log to read afterward (see the diagnostics section above); the homelab variant additionally enables SSH on the live installer itself (root, same key as the installed host — anyone holding this ISO already has full disk access anyway), reachable as nixos-unattended.local via avahi. From another machine on the same network:

ssh root@nixos-unattended.local
tail -f /tmp/unattended/install.log

/tmp/unattended/install.log is the file the script tees to continuously — the copy that later lands in /var/log/unattended/ on the target disk (and on the NIXLOG USB partition, if present) is only a snapshot taken at a few checkpoints, not a live stream. Boot only one homelab-n stick at a time — all of them announce the same live-installer hostname (nixos-unattended), so more than one on the network at once makes nixos-unattended.local ambiguous.


Nitrokey SSH (OpenPGP smartcard)

Nitrokey SSH is enabled by importing modules/nixos/nitrokey.nix (currently on tux and the latitude-7212 tablet): hardware.nitrokey.enable (udev), services.pcscd.enable (smartcard daemon), and programs.gnupg.agent with enableSSHSupport — so gpg-agent is the SSH agent and the card's authentication subkey signs SSH challenges. modules/fish.nix already points SSH_AUTH_SOCK at gpg-agent's socket. The PIN prompt is the customNitrokey.pinentryPackage option: terminal pinentry-curses by default (tux); the touch tablet overrides it to pinentry-gnome3 so the PIN dialog is usable by touch and triggers the on-screen keyboard. Using a second Nitrokey on another host just needs the same auth subkey provisioned onto it (below) and, if it's a different key, its gpg --export-ssh-key public key added to the git server.

You only ever export the public key; the private keys never leave the card.

Provisioning a key on a Nitrokey 3 (OpenPGP applet)

Always back up the secret key (step 3) before keytocard — it's a move. Losing the backup (and having no fetch URL / keyserver copy) means GPG can never use the card keys again.

# 1. Reset the OpenPGP applet (wipes old keys, unblocks PINs; other applets untouched)
gpg --card-edit → admin → factory-reset      # defaults: user 123456 / admin 12345678

# 2. Generate a certify-only primary + S/E/A subkeys (RSA 4096)
gpg --expert --full-generate-key             # (8) RSA set-own-caps -> Certify only, 4096
gpg --expert --edit-key <email>              # addkey ×3: sign-only, encrypt-only, auth-only; save

# 3. BACK UP FIRST (offline!)
gpg --export-secret-keys --armor <email> > gpg-secret-backup.asc
gpg --export             --armor <email> > gpg-public.asc

# 4. Move subkeys to the card
gpg --edit-key <email>                       # key 1/2/3 -> keytocard (Sig/Enc/Auth); save

# 5. Set a fetch URL (host gpg-public.asc somewhere) + change PINs
gpg --edit-card → admin → url / passwd

# 6. Export the SSH public key for your VCS / git server
gpg --export-ssh-key <email> > nitrokey.pub

Using it on another machine

Insert the card, then get the public key into that keyring:

gpg --edit-card → fetch      # if a URL is set (step 5); else: gpg --import gpg-public.asc
gpg --card-status            # "General key info" now shows your key
ssh-add -L                   # card's SSH key, served by gpg-agent

Optional — sign commits with it: git config --global commit.gpgsign true and git config --global user.signingkey <primary-keyid>.

Recovering an SSH pubkey when the GPG certificate is lost but the card still has the keys: read it straight off the card over PKCS#11 — gpgconf --kill scdaemon; ssh-keygen -D opensc-pkcs11.so.


Desktop bar — waybar / quickshell (tux)

Two bar implementations are always installed side by side on hosts that import modules/home/desktop (currently tux): modules/home/desktop/waybar.nix (stable) and modules/home/desktop/quickshell.nix (WIP QtQuick-based rebuild, config in configs/quickshell/). Only one auto-starts via systemd at a time — set with custom.desktopBar = "waybar" | "quickshell"; on a host's home.nix (defaults to "waybar"; tux currently sets "waybar" explicitly, per hosts/tux/home.nix). Since both are installed regardless, switching without a rebuild is just systemctl --user start/stop waybar quickshell.

quickshell mirrors waybar's modules (workspaces, clock, cpu/mem/disk/temp/backlight, network, bluetooth, volume, battery, peripheral batteries, weather, webcam toggle, tray) and adds a Wi-Fi picker popup (nmcli-backed — see below), a native Quickshell.Bluetooth popup, and its own session/power menu (lock/suspend/hibernate/reboot/shutdown/logout — PowerMenu.qml; waybar has the same actions via wlogout instead, launched from a custom/powermenu button, since waybar has no native popup widget to build one with). The RAPL wattage gauge got an inline sparkline instead of a popup. The mail-unread widget was dropped from both (see "Mail" below) and the calendar-grid clock tooltip wasn't rebuilt — quickshell's clock just shows a plain date/time tooltip.

Gotcha: stale QML disk cache. Qt caches compiled QML bytecode across runs, and it doesn't reliably invalidate across this config's restart-heavy WIP edit/rebuild cycle — a widget can keep showing pre-edit behavior (e.g. an icon that was empty in an earlier version stays empty even though the source now has it) with zero errors in the logs. quickshell.nix sets QML_DISABLE_DISK_CACHE=1 on the service specifically to avoid this — remove that override once the config has stabilized, since the cache is a legitimate startup-time win otherwise. If it's ever re-enabled and this bites again, clear it by hand:

rm -rf ~/.cache/qmlcache
systemctl --user restart quickshell

Wi-Fi in quickshell shells out to nmcli (configs/quickshell/scripts/wifi-*.sh) rather than using the native Quickshell.Networking module — that module's QML property/method names couldn't be verified against the pinned nixpkgs quickshell build at the time this was written (quickshell.org's type docs 403 every fetch; only Forgejo source mirrors and search-engine snippets were usable), and a wrong guess there breaks the whole panel. Bluetooth and UPower/Pipewire are native, verified against the upstream C++ headers. If revisiting this, re-check Quickshell.Networking's actual API first.


Mail — local-first (mbsync + notmuch + aerc/meli) + KMail/Claws Mail/Betterbird

modules/mail.nix wires up a local-first mail stack to test against GUI clients: mbsync (isync) pulls IMAP down to ~/Mail/<account> on a 5-minute systemd timer, notmuch indexes it (Xapian full-text search, scales to huge mailboxes without the lag Betterbird hit), and aerc/meli are terminal clients on top. kdePackages.kmail is installed as a GUI alternative (home.nix); Claws Mail and Betterbird are installed declaratively via Flatpak (modules/nixos/desktop.nix → services.flatpak.packages). There is no bar widget for unread count any more — it was removed from waybar (see "Desktop bar" below) and hasn't been rebuilt in quickshell.

Credentials come from a self-hosted Vaultwarden instance via rbw (an unofficial but scriptable CLI — unlike the official bw, it keeps an unlocked vault behind a background agent, so passwordCommand doesn't need to prompt on every sync). programs.rbw.settings.base_url in modules/mail.nix points at that instance, so rbw never talks to bitwarden.com.

Accounts (accounts.email.accounts in modules/mail.nix): julius@zeidlos.com (Gmail/Google Workspace, primary) plus several kasserver.com-hosted IMAP addresses (jz@chaos.li, banking@zeidlos.de, do7jz@chaos.li). Each account's passwordCommand pulls its password from a matching Bitwarden item — for Gmail, that item must hold an App Password (requires 2FA), not the account's normal login password.

One-time setup:

  1. rbw login and rbw unlock once, interactively (uses base_url/email already set in modules/mail.nix, so no rbw config set needed first).
  2. sudo nixos-rebuild switch --flake ~/Projects/nix/nixconfigs#tux, then run mbsync --all once by hand to do the initial sync (the systemd timer only fires every 5 minutes after that).

mbsync's remove/expunge default to none, so nothing propagates deletions in either direction while this is just a test setup.

Adding another account: scripts/gen-mail-accounts.sh reads a Bitwarden folder (default email) via rbw and prints ready-to-paste accounts.email.accounts.<name> blocks matching the ones already in modules/mail.nix — it assumes item names look like <address>-password / <address>-app-password, that the item's Bitwarden username field holds the provider login (e.g. a kasserver.com account number, not the email address), and that every non-Gmail domain shares one IMAP/SMTP host. No password ever appears in its output; review the printed block before pasting it in.

rbw unlock
nix shell nixpkgs#jq nixpkgs#rbw -c ./scripts/gen-mail-accounts.sh

Brave — de-Googled, privacy-first Chromium (with Web Bluetooth)

A Chromium browser for the sites Zen/Firefox can't handle, stripped of the Google/crypto/AI bloat. Brave is installed per-user via home-manager (modules/home/desktop/brave.nix → programs.brave); the debloat is enforced by a locked managed policy (configs/brave-policies.json) that force-disables Rewards, Wallet, VPN, Leo AI, News, Talk and Tor, plus all telemetry (P3A, stats ping, Web Discovery, metrics reporting) and Sync/Safe Browsing/autofill. Policy keys set there are greyed out in settings — verify at brave://policy.

Chromium reads managed policies from /etc/brave/policies/managed/*.json, shipped declaratively by modules/nixos/brave.nix (environment.etc), imported via modules/nixos/desktop.nix so every NixOS desktop host (tux, big-rig) gets it — no manual steps needed.

Web Bluetooth is off by default in Linux Chromium, so programs.brave.commandLineArgs launches Brave with --enable-experimental-web-platform-features; BlueZ is already enabled system-wide (modules/nixos/common.nix). That switch turns on all experimental web-platform features — if --enable-features=WebBluetooth alone suffices on your Brave build, prefer the narrower flag.


Neovim config

The LazyVim config lives at configs/nvim/ in this repo. modules/neovim.nix points ~/.config/nvim at it via an out-of-store symlink (not a store copy), so LazyVim can write lazy-lock.json back into the repo on :Lazy update/sync/clean — commit that file's changes like any other edit.


Claude Code — nixos MCP server

mcp-nixos is a read-only MCP server that lets Claude Code query NixOS packages/options, Home Manager, nix-darwin, flakes, and package version history. It's registered project-scoped in .mcp.json, launched via uv so it also works in sandboxes/containers that don't have the home-manager profile (the PyPI package is fetched on demand):

{ "mcpServers": { "nixos": { "command": "uv", "args": ["tool", "run", "mcp-nixos"] } } }

(uv tool run is the long form of uvx; the Nix package mcp-nixos in modules/home/default.nix still provides the CLI directly on hosts. It's temporarily taken from the nixpkgs-mcp-nixos input, pinned to the nixpkgs before the 2026-10-23 update, because newer nixpkgs marks its fastmcp-slim dependency broken; drop the pin once nixpkgs ships fastmcp 4.x.) On first launch in this repo, Claude Code prompts to approve the server; use /mcp to reconnect after config changes. It needs outbound network (PyPI on first run, then search.nixos.org, FlakeHub, noogle.dev, cache.nixos.org) at runtime.

Claude Code's /voice dictation records the mic via SoX, which the CLI looks for on $PATH; sox is in modules/home/desktop/default.nix (the graphical desktop clients only — servers have no mic and don't need it).


Monitoring — pushing host metrics to chaos-cloud

modules/nixos/monitoring.nix (customMonitoring.enable) runs node_exporter plus a Prometheus instance in agent mode, pushing this host's CPU/RAM/disk metrics to the central Prometheus in chaos-cloud (https://metrics.cloud.zeidler.dev) — same push-based pattern as pangolin's and datengrab's metrics-agent, since these hosts aren't reachable from chaos-cloud to be scraped. Standard for every server (each host importing modules/nixos/server.nix: homelab-1/2/3, hetzner-robot-1): server.nix turns on both customMonitoring and customLogs (below) with usernames derived from the hostname (<host>-writer, <host>-logs-writer) and password files at /etc/nixos-secrets/{metrics,logs}-password. Clients (tux, big-rig, latitude-7212) don't ship anything. A new server needs both users added to chaos-cloud's metrics-write-auth / logs-write-auth secrets, a LogsMissing rule in loki/log-shipping-rules.yaml, and its instance in the disk/memory alert regexes; HostDown and the infra-overview dashboard pick it up by themselves.

Logs: modules/nixos/logs.nix (customLogs) runs Grafana Alloy shipping the systemd journal to chaos-cloud's Loki (https://logs.cloud.zeidler.dev), labels cluster=<host>, job=journal, unit=<systemd unit>, no replay older than 6h. Alloy runs as a DynamicUser, so the password reaches it via systemd LoadCredential instead of a file it would need to read itself.

Fritz!Box: modules/nixos/fritzbox.nix (customFritzbox, imported and enabled on homelab-2 only) runs fritz-exporter (TR-064) on localhost and adds a fritzbox job to the metrics agent via customMonitoring.extraScrapeConfigs, so the router's metrics (WAN traffic, line, Wi-Fi) go out through the same remote_write as instance=fritzbox. Needs, by hand: TR-064 enabled on the box ("Zugriff für Anwendungen zulassen"), a dedicated FRITZ!Box user (default name metrics) and its password in /etc/nixos-secrets/fritzbox-password (root-only, handed over via LoadCredential). When the line is down nothing arrives in chaos-cloud; the gap itself is the signal. The exporter unit uses Restart=always/RestartSec=60 with the start-limit disabled, so it keeps retrying while the router is unreachable (host boot, Fritz!Box firmware reboot) instead of giving up and staying failed.

Each host needs its own remoteWriteUsername (set per-host, matches an entry in chaos-cloud's metrics-write-auth-secret.yaml) and a passwordFile — an absolute path to a plain root-only file containing just the raw password (no KEY=value, unlike the Wi-Fi/Pangolin .env secrets). Prometheus reads it directly at runtime (password_file:), so it's never in the Nix store. Uses the same secretFile-style convention as customPangolin (/etc/nixos-secrets/metrics-password) rather than sops — only tux has a sops recipient registered at all (see .sops.yaml), and this was kept uniform rather than special-casing just that one host.

Known limitations / things installed out-of-band

Package Reason
zen-browser Not in nixpkgs — NixOS hosts get it via declarative Flatpak (modules/nixos/desktop.nix → services.flatpak.packages)
betterbird Not in nixpkgs — NixOS hosts get it via declarative Flatpak (modules/nixos/desktop.nix)
krew kubectl plugin manager — install manually

Historical note: while big-rig ran CachyOS/Arch (before its 2026-08-30 conversion to NixOS), this table also listed hyprlock (nixpkgs' build links against the NixOS PAM path — use AUR on Arch), cachyos-fish-config and keyd (both CachyOS/pacman-specific), and virtualbox-host-dkms (sudo pacman -S virtualbox-host-dkms linux-cachyos-headers && sudo modprobe vboxdrv — CachyOS's custom kernel needed DKMS since Nix can only install the userspace virtualbox binary). None of that applies anymore: NixOS's hyprlock build is fine under NixOS's own PAM path, there's no CachyOS/pacman on this host anymore, kanata replaces keyd, and VMs on big-rig are libvirt/QEMU via virt-manager (desktop.nix, same as every other desktop host) rather than VirtualBox at all now.


Ham radio

Amateur-radio software is bundled in one reusable home-manager module, modules/home/ham-radio.nix, imported by every host that carries a radio — currently tux (shack) and the rugged latitude-7212 tablet (field). Both get the identical set; edit the list once in the module rather than per host. It's a "Ham Radio Linux"-style toolkit grouped by function:

Function Tools
Weak-signal / keyboard digital js8call, wsjtx (FT8/FT4/WSPR), fldigi + flrig
APRS / packet direwolf (soundcard TNC), xastir (map client), aprx (iGate/digipeater), multimon-ng, ax25-tools, ax25-apps
Rig control hamlib (rigctl/rigctld), wfview (GUI control/waterfall for modern Icom rigs)
Logging / QSL qlog, tqsl (LoTW)
Winlink pat (email over radio)
Satellites gpredict
Image / CW qsstv (SSTV), aldo (CW trainer)
SDR receive / DSP gqrx, sdrangel, cubicsdr, gnuradio (GRC flowgraphs)
Radio programming chirp (analog), qdmr (DMR)

gridtracker (WSJT-X/JS8Call map companion) was dropped: it bundles nwjs, which nixpkgs marks insecure (Chrome 139.0.7258.128, known unpatched vulnerabilities) — rather than add a permittedInsecurePackages override, we drop the package to avoid pulling in a known-insecure bundled runtime.

Everything is a plain user package — no system services are enabled by default. USB-serial CAT/rig control and radio programming work without root because jz is in the dialout group (modules/nixos/common.nix). direwolf/aprx are installed as CLI tools; run them by hand per session, or promote to a NixOS service later if you want a persistent iGate/digipeater. ax25-tools needs overlays/ax25-tools.nix to build at all: its hdlcutil/ subdir (sethdlc/smdiag/smmixer, config tools for the long-removed hdlcdrv/baycom/soundmodem kernel drivers) includes a kernel header, linux/hdlcdrv.h, that current nixpkgs headers no longer ship — the overlay drops that subdir from the build (dead code for hardware no modern kernel drives anyway) rather than vendor the removed header; drop the overlay once nixpkgs' ax25-tools does the same upstream.

The NixOS-side counterpart, modules/nixos/ham-radio.nix (imported by the same two hosts), sets hardware.hackrf.enable = true and puts jz in plugdev so HackRF-family USB SDRs — including the rad1o badge, which enumerates as a HackRF (1d50:6089) — work in gqrx/hackrf_info without root. Without it the device node stays root-owned and gqrx fails with a permissions error.

angryoxide is a Wi-Fi (802.11) attack tool, not ham radio — it lives in the security module (modules/home/security.nix), not this one.


Reticulum (LoRa mesh)

Reticulum is set up on tux, latitude-7212, homelab-1, homelab-2 and hetzner-robot-1 for testing MicroTAK RF sync over LoRa. Two modules:

  • modules/nixos/reticulum.nix (customReticulum) runs the stock services.rnsd daemon as an always-on shared instance (transport node). It declares: an RNode LoRa interface on /dev/rnode0 (EU 868 MHz defaults — lora.{frequency,bandwidth,spreadingFactor,codingRate,txPower}), an AutoInterface LAN bonus path, and a 127.0.0.1:4242 TCP bridge for the login user's apps. panic_on_interface_error = No keeps rnsd up with no radio attached; there is no health check (its rnstatus probe could take >120s on a busy multi-NIC host like tux and fail an otherwise-fine daemon).
  • Transport topology — explicit TCP, not multicast. AutoInterface (IPv6 multicast discovery) is unreliable here (Wi-Fi homelab boxes, k8s overlay NICs, and the nixpkgs module opens the wrong firewall port — 27916 vs RNS's actual 29716, which this module corrects). So hosts peer over explicit TCP via hub.serveLan / hub.servePublic / uplinks:
    • hetzner-robot-1 — public hub (public IP, always on) so roaming tux/latitude reach the mesh off-LAN;
    • homelab-1 — LAN hub, and also dials the public hub so LAN and off-site are one network;
    • homelab-2 dials the LAN hub; tux/latitude dial both (LAN at home, public when away). Off-grid, the LoRa RNode carries the mesh with no IP at all.
  • IFAC (private network). Every mesh-facing interface carries an Interface Access Code (ifac.network name + a shared passphrase), so only our nodes join — important because the public hub's port is open to the internet (RNS payloads are E2E-encrypted regardless; IFAC gates the transport layer). The passphrase is a secret kept out of the Nix store: it lives in /etc/nixos-secrets/reticulum-ifac (root 0600, same on every host) and is loaded as a systemd credential and spliced into the generated config at service start — the store config carries only a placeholder.
  • modules/home/reticulum.nix (customReticulum, home-manager) installs nomadnet (the TUI) and the rns CLIs (rnstatus, rnpath, rnprobe, rncp, rnx, rnodeconf), and writes ~/.reticulum/config as a small endpoint node peering to the local rnsd over the :4242 bridge — the apps never open the radio themselves (rnsd is a locked-down DynamicUser whose instance socket the login user can't share). On the headless hosts, SSH in and run nomadnet.

Manual, per host: the IFAC passphrase file must exist before rnsd starts (comin hosts get it pre-placed; on tux/latitude install the same bytes to /etc/nixos-secrets/reticulum-ifac, root 0600). Per board: flash RNode firmware over USB with rnodeconf --autoinstall (from rns) — not declarable in Nix — then confirm the board's USB VID:PID with lsusb: the udev rules carry likely IDs for the Heltec V3 (CP2102/CH9102) and T-Echo (nRF52840), but these vary by revision and may need editing in modules/nixos/reticulum.nix (10c4:ea60 is a generic serial ID — keep it only where the RNode is the sole CP2102 device).


Embedded/firmware dev toolchain

modules/home/embedded-dev.nix, imported by tux and big-rig: gcc-arm-embedded (arm-none-eabi-gcc cross toolchain for Cortex-M/R), gnumake, cmake, tinyxxd (standalone xxd — hex dump / xxd -i C-array conversion — without pulling in vim), and a python3 with pyyaml importable. Generic ARM firmware build toolchain, not tied to any one project.


Security tooling

Two home-manager modules carry the DevSecOps / security-research toolkit. The base image (modules/home/default.nix, every host) already ships grype/trivy/syft for SCA/SBOM; these modules layer on the rest. RF signal research is split from appsec so the two can be imported independently — and they are: the heavy pentest suite (security.nix) is imported on tux only (kept off the low-RAM latitude-7212 tablet), while the RF/SDR field kit (rf-sec.nix) rides along with the SDR gear on both tux and latitude-7212.

modules/home/security.nix — DevSecOps + pentest (tux only):

Area Tools
Secrets trufflehog, gitleaks
IaC / misconfig trivy config (base image); checkov dropped — hard dep on insecure ecdsa
Supply-chain signing cosign (Sigstore)
Vuln / dependency scan vulnix, osv-scanner
Cloud & K8s kubescape, kube-bench, cloudhunter, prowler, scoutsuite
Exploitation frameworks metasploit; routersploit dropped — unmaintained, imports pkg_resources (removed in Python 3.14)
Passwords / hashes hashcat, john, thc-hydra
Recon / attack surface amass, subfinder, dnsx, naabu, httpx, katana
Web / DAST nuclei (+nuclei-templates), nikto, ffuf, feroxbuster, sqlmap, testssl
Network & traffic nmap, zmap, wireshark, netsniff-ng, dsniff, bettercap, mitmproxy, snort (appid link fix via overlays/snort.nix), ssh-audit
Wireless angryoxide, aircrack-ng (Wi-Fi 802.11)
Tunneling / anonymity iodine (IP-over-DNS), tor-browser

modules/home/rf-sec.nix — RF/SDR signal research (the analysis layer above the plain SDR receivers in ham-radio.nix): urh (Universal Radio Hacker — demod/decode/fuzz), inspectrum (offline signal analysis), rtl_433 (ISM device decode), kalibrate-rtl (ppm calibration), soapysdr (SDR abstraction). Hardware-specific host tools (hackrf, airspy, ubertooth, killerbee, proxmark3) are listed commented-out in the module — uncomment per the gear you own.

Live capture (wireshark), and raw nmap/bettercap modes, need elevated privileges. Run them under sudo, or enable the NixOS programs.wireshark module for a setcap'd dumpcap if you want non-root capture.


Custom packages — flake outputs

All packages under pkgs/ are exposed as top-level flake outputs for direct builds and CI:

nix build .#aria2tui
nix build .#bookokrat    # resolves to upstream flake input (github:bugzmanov/bookokrat)
nix build .#nmrs

bookokrat comes from an upstream flake input — update it with nix flake update bookokrat. No local package. It currently needs a one-line postPatch (in both flake.nix's packages.bookokrat and hosts/tux/home.nix's inline overrideAttrs, since they're separate overrides of the same input): upstream's src/clipboard.rs fn detect() -> Provider has a #[cfg(not(target_os = "android"))] block that falls off the end without a return, a genuine compile break on any non-android target (not a Nix issue) — same fix rustc itself suggests. Drop the postPatch once upstream fixes it.

eilmeldung used to be its own upstream flake input; it moved into nixpkgs unstable (pkgs.eilmeldung) and its home-manager module into home-manager's own repo, so hosts/tux/home.nix now just lists it as a plain pkgs package like any other nixpkgs entry — no flake input, no custom output, no nix flake update step for it.


Supply chain security

CI workflows (.forgejo/workflows/)

Workflow Trigger What it does
security-scan.yml push to main, weekly Monday 06:00 UTC Builds the custom packages (aria2tui, bookokrat, nmrs); runs vulnix on each runtime closure (-C) against ci/vulnix-whitelist.toml so it's red only on a new/changed CVE, plus grype (fail-on-HIGH, non-blocking); generates CycloneDX SBOMs uploaded as 90-day artifacts (upload-artifact@v3 — v4 needs the GHES-only backend)
update-flake.yml nightly 04:00 UTC The hmu canary: nix flake update, then builds every host (all nixosConfigurations + big-rig's home config). If any fails the run is red and no PR opens; green opens a PR with a table of updated input revisions, so a pull+hms (tux/latitude) or merge (servers via comin) is known-safe
deploy-gate.yml push to main Builds homelab-1/2/3 and hetzner-robot-1, then fast-forwards the deploy branch that comin hosts follow (non-force push)

The update and deploy-gate workflows require a FORGEJO_TOKEN secret (repo → Settings → Secrets) — a personal access token with repository write + pull-request write scope.

Ad-hoc scanning

# CVE scan a built package
OUT=$(nix build .#bookokrat --print-out-paths --no-link)
nix run nixpkgs#vulnix -- "$OUT"

# Generate a CycloneDX SBOM  (syft is installed via modules/common.nix)
syft scan dir:"$OUT" --output cyclonedx-json --file bookokrat.cdx.json

Structure

nixconfig/
├── flake.nix          # inputs: nixpkgs-unstable + home-manager
├── arch.nix           # CachyOS/Arch (x86_64-linux): all modules + Linux packages
├── nixos-vm.nix       # NixOS VM (x86_64-linux): trimmed desktop-essentials home-manager profile
├── macbook.nix        # macOS (x86_64-darwin + aarch64-darwin): shared mac config
├── hosts/
│   └── nixos-vm/
│       ├── configuration.nix       # NixOS system config: bootloader, networking, users
│       └── hardware-configuration.nix  # generated by nixos-generate-config, machine-specific
├── modules/
│   ├── fish.nix           # shell, aliases, abbreviations, functions
│   ├── git.nix            # git + lazygit
│   ├── tmux.nix           # tmux with nixpkgs plugins (replaces TPM)
│   ├── neovim.nix         # neovim binary + LSP deps (LazyVim manages plugins)
│   ├── starship.nix       # prompt
│   ├── terminal.nix       # ghostty
│   ├── herdr.nix          # herdr config.toml + HERDR_PROCESS_DETECTION
│   ├── waybar.nix         # waybar config + style — shared by arch.nix and home.nix (Linux only)
│   ├── mail.nix           # mbsync/notmuch/aerc/meli — local-first mail (see "Mail" section)
│   ├── nixos/
│   │   ├── desktop.nix    # NixOS system module: hyprland, pipewire, polkit, thunar, tty1 autologin
│   │   ├── npu.nix        # AMD XDNA2 NPU device access (render group)
│   │   └── streamdeck.nix # Elgato Stream Deck: StreamController + udev rules + user service
│   └── arch/
│       ├── hyprland.nix   # hyprland + hyprpaper + hypridle  (Linux only)
│       ├── desktop.nix    # rofi, dunst, yazi, btop, xdg portals  (Linux only)
│       ├── logiops.nix    # MX Master 4 logiops config + systemd service  (Linux only)
│       └── media.nix      # mpd, ncspot                      (Linux only)
├── .forgejo/workflows/ # CI: security-scan.yml (vulnix/grype/sbom), update-flake.yml (nightly build-check + PR), deploy-gate.yml (→ deploy branch)
├── pkgs/              # custom packages not in nixpkgs, callPackage'd from modules
│   ├── adsb-tui/      # terminal ADS-B aircraft tracker, packaged from upstream release tarball
│   ├── aria2tui/      # Python TUI frontend for aria2c, packaged from PyPI
│   # bookokrat — upstream flake input (github:bugzmanov/bookokrat); no local package
│   ├── docker-sbx/    # Docker Sandboxes CLI, packaged from upstream release tarball
│   └── nmrs/          # waybar network on-click GUI, built from networkmanager-rs/nmrs-gui
└── configs/           # raw config files referenced by modules
    ├── hyprland.lua
    ├── hyprlock.conf
    ├── dunstrc
    ├── logid.cfg
    ├── kanata.kbd
    ├── waybar-style.css
    ├── waybar/
    │   ├── webcam.sh
    │   ├── timezones.sh
    │   ├── mx4-battery.sh
    │   └── corne-battery.sh
    └── quickshell/        # WIP waybar replacement — see "Desktop bar" section
        ├── shell.qml
        ├── Bar.qml
        ├── Theme.qml / Pill.qml / Label.qml / JsonPoller.qml
        ├── modules/        # one file per bar module
        ├── popups/         # WifiPanel, BluetoothPanel, PowerMenu
        └── scripts/        # nmcli/upower/sysfs helpers, JSON-out like configs/waybar/*.sh