Skip to content
cobalt

Runbook — VM rehearsal

Status: run on 2026-10-17 — three full rounds under TCG on an Apple-silicon Mac (install with --flake, offline install with the guest’s network cut, whole-disk NVMe layout), each followed by the first boot; all three passed. It exercises scripts/install.sh and the first boot in a virtual machine; the real procedure is the installation runbook. Nothing here touches a physical disk.

What is emulated

Section titled “What is emulated”
Real hardware (install.md)In the VM (scripts/vm-rehearsal.sh)
key stick (≥ 32 GB, USB)vm/usb.qcow2, 32 GB sparse, USB mass storage on an xHCI controller → /dev/disk/by-id/usb-QEMU_QEMU_HARDDISK_cobaltvmusb-0:0 (expected name)
WD SN740 NVMevm/nvme.qcow2, 64 GB sparse, NVMe controller, 512e sectors → /dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_cobaltvmnvme (expected name)
temporary installer stickthe ISO as a SATA CD-ROM (install mode only)
Insyde UEFI firmware + NVRAMOVMF (EDK2) + vm/efivars.fd, a writable copy of its variable store
YubiKeynone — or the canokey emulated FIDO2 token if QEMU was built with it (§3)
Wi-Fiuser-mode NAT on a virtio NIC; the host is 10.0.2.2 from inside the guest
consoleserial console + QEMU monitor on the terminal, plus the normal graphics window

The VM gets 4 GB RAM and 4 CPUs. On a Linux host with /dev/kvm it runs with KVM; on an Apple-silicon Mac an x86_64 guest can only be interpreted (-accel tcg): expect every step to take ten to thirty times longer than on the laptop — nixos-install copying the embedded closure is a matter of hours, not minutes.

What you need

Section titled “What you need”
  • QEMU with qemu-system-x86_64 and qemu-img: nix shell nixpkgs#qemu (the Android SDK also ships a qemu-img; make sure the one on PATH is QEMU’s, or set COBALT_VM_QEMU_IMG).
  • The installer ISO (§1), built on a host that can build x86_64-linux derivations.
  • The OVMF firmware files (§2).
  • About 20 GB of free space: the ISO (several GB with the closure embedded) plus whatever the guest writes into the two sparse images.
  • Optional: QEMU with canokey support for the FIDO2 steps (§3).

1. Build the installer image

Section titled “1. Build the installer image”
Terminal window
nix build path:.#images.installer      # → result/iso/cobalt-installer.iso

image.baseName is forced to cobalt-installer, so the file is result/iso/cobalt-installer.iso (no version infix). On macOS this needs the nix.linux-builder VM with x86_64 emulation enabled or any x86_64-linux machine with Nix; the closure is large, so a first build is slow.

Device ids. The ISO embeds /etc/cobalt/config including my/private.nix, and install.sh only asks for device ids when that file is absent. With the laptop’s ids embedded the script would stop at USB key not found. Two ways to rehearse:

  • A — build the ISO for the VM (recommended). Set disks in my/private.nix to the VM names (table above; confirm them in the guest first if in doubt, §5), build the ISO, then restore the real ids. The embedded closure then matches the VM exactly and the install is fully offline.
  • B — reuse the hardware ISO. Inside the guest, copy the configuration, edit the ids, and point the installer at the copy: cp -r /etc/cobalt/config /tmp/cfg && chmod -R u+w /tmp/cfg, edit /tmp/cfg/my/private.nix, then run the installer with COBALT_CONFIG=/tmp/cfg. The device path only feeds disko (filesystems are declared by partlabel), so the system closure should be unchanged; if nixos-install starts building instead of copying, the closure differs and the VM needs network and time.

Whatever identity data my/private.nix carries ends up on the virtual disks and in the ISO. Use placeholders for the rehearsal if that matters, and delete vm/ and the ISO afterwards.

Passwords. Accounts are declarative; without secrets/users.yaml decryptable by the VM’s host age key nobody can log in after the first boot (modules/nixos/users.nix). For a complete rehearsal generate a throwaway host key on the workstation (age-keygen -o vm-host-age.txt), add its public key to .sops.yaml as &cobalt, run sops updatekeys secrets/*.yaml, build the ISO, and revert the recipient afterwards. The key file is handed to install.sh in §5 — never put it into the repository tree, and never reuse the laptop’s real host key for this.

2. Firmware (OVMF)

Section titled “2. Firmware (OVMF)”
Terminal window
nix build nixpkgs#OVMF.fd -o vm/ovmf                               # x86_64-linux host
nix build nixpkgs#legacyPackages.x86_64-linux.OVMF.fd -o vm/ovmf   # macOS: fetched from the binary cache

The fd output contains FV/OVMF_CODE.fd (firmware) and FV/OVMF_VARS.fd (variable template). The script passes them exactly as nixos/modules/virtualisation/qemu-vm.nix does: -drive if=pflash,format=raw,unit=0,readonly=on,file=…/OVMF_CODE.fd and -drive if=pflash,format=raw,unit=1,readonly=off,file=vm/efivars.fd, where efivars.fd is the copy that prepare makes. On an aarch64 Mac nixpkgs#OVMF is the AArch64 firmware (AAVMF) — not usable for this guest; the legacyPackages.x86_64-linux attribute is served by cache.nixos.org, so no builder is needed.

Secure Boot. The plain OVMF build is compiled without SECURE_BOOT_ENABLE; bootctl status in the guest then reports Secure Boot as unsupported, and sbctl enroll-keys has nothing to enroll into. To rehearse step 4 of install.md use OVMFFull.fd instead (built with Secure Boot and SMM; its OVMF_VARS.fd holds no keys, i.e. the firmware starts in Setup Mode — pick OVMF_VARS.fd, not OVMF_VARS.ms.fd, which already has Microsoft’s keys enrolled). The script detects such a build from the firmware descriptor shipped next to it (requires-smm) and adds -machine q35,smm=on plus -global driver=cfi.pflash01,property=secure,value=on (edk2 OvmfPkg/README, “SMM support”); SMM under TCG makes the firmware noticeably slower.

3. QEMU, and the optional emulated FIDO2 token

Section titled “3. QEMU, and the optional emulated FIDO2 token”

nixpkgs#qemu has no canokey device. QEMU built with qemu.override { canokeySupport = true; } (what nixos/tests/systemd-initrd-luks-fido2.nix uses) adds a CanoKey emulator — a virtual FIDO2/OpenPGP card, not a YubiKey. This is a from-source build of QEMU; it has not been tried on macOS. For example:

Terminal window
nix build --impure --expr \
  '(import <nixpkgs> {}).qemu.override { canokeySupport = true; }' -o vm/qemu
COBALT_VM_QEMU=vm/qemu/bin/qemu-system-x86_64 ./scripts/vm-rehearsal.sh install

When the device exists the script attaches it exactly as the nixpkgs test does (-device pci-ohci,id=usb-bus -device canokey,bus=usb-bus.0,file=vm/canokey-state); the state file is created on first use and must stay the same for install and boot. Without it the script prints a notice and the installer must be run with COBALT_SKIP_FIDO2=1.

4. Prepare

Section titled “4. Prepare”
Terminal window
./scripts/vm-rehearsal.sh prepare result/iso/cobalt-installer.iso

Creates vm/usb.qcow2 (32 GB) and vm/nvme.qcow2 (64 GB) as sparse qcow2 images, copies OVMF_VARS.fd to vm/efivars.fd (empty NVRAM: no boot entries, Setup Mode) and records the ISO path. Running prepare again asks before erasing a previous rehearsal (COBALT_VM_FORCE=1 skips the question). vm/ is gitignored; COBALT_VM_DIR moves the work dir elsewhere.

5. Install inside the VM

Section titled “5. Install inside the VM”
Terminal window
./scripts/vm-rehearsal.sh install

  1. Boot menu. The ISO’s GRUB menu appears in the graphics window (unless COBALT_VM_UKI boots the installer UKI directly — §5b). To work on the terminal (paste-able), open Options → Serial console=ttyS0,115200n8 and choose the NixOS entry there; the kernel then also talks on the serial port and a login shell appears on this terminal. The plain entry works too — then everything happens in the window.

  2. Device names. ls -l /dev/disk/by-id/ — the two names should match the table above. If they do not, use the names shown (and, for option A, rebuild the ISO with them).

  3. Files for the installer. A prepared my/private.nix (option B) and the throwaway host age key can be pasted over the serial console with a heredoc, or served from the host (python3 -m http.server 8000 in a scratch directory; in the guest curl -O http://10.0.2.2:8000/vm-host-age.txt).

  4. Run the installer — as root, without a token:

    Terminal window
    sudo COBALT_SKIP_FIDO2=1 /etc/cobalt/install.sh
    # option B: sudo COBALT_SKIP_FIDO2=1 COBALT_CONFIG=/tmp/cfg /etc/cobalt/install.sh

    With the canokey device present, drop COBALT_SKIP_FIDO2=1: systemd-cryptenroll --fido2-device=auto then enrolls the emulated token on the detached header, the way the nixpkgs test does.

  5. Prompts, in order: path to my/private.nix (only if the config has none) · path to the host age key (empty generates one — fine for the rehearsal) · the NVMe and USB by-id paths typed exactly · the LUKS passphrase (the recovery fallback) · the generated recovery key is printed (note it: it is one of the two ways to unlock in the VM) · a path for the LUKS header backup (/tmp/header.img is enough here; it is lost at power-off).

  6. Then watch for: disko partitioning the USB disk before the NVMe and formatting the LUKS data area with the header on disk-a-usb-reserved · /persist seeding and sbctl create-keys · nixos-install copying from the embedded closure without any download (under TCG this is the long part) · the iso9660 byte copy into the key’s installer partition (isosize on /dev/sr0) · the signed UKI landing at EFI/Linux/cobalt-installer.efi.

  7. poweroff the guest when the script prints done.

5b. Unattended run — serial console on named pipes

Section titled “5b. Unattended run — serial console on named pipes”

For a rehearsal driven by a script (or by an agent on the host) the harness can boot the installer UKI directly and put the serial console on two FIFOs instead of the terminal:

Terminal window
mkfifo vm/serial.in vm/serial.out
cat vm/serial.out >> vm/serial.log &          # keep a reader attached for the whole run
COBALT_VM_HEADLESS=1 COBALT_VM_OFFLINE=1 COBALT_VM_SERIAL=pipe:$PWD/vm/serial \
COBALT_VM_UKI=result-uki/cobalt-installer.efi COBALT_VM_CMDLINE_EXTRA='console=ttyS0,115200n8' \
./scripts/vm-rehearsal.sh install
printf 'ls -l /dev/disk/by-id/\n' > vm/serial.in               # type into the guest
printf 'screendump vm/screen.ppm\n' | nc -U vm/monitor.sock      # QEMU monitor (also: sendkey, quit)
  • COBALT_VM_UKI skips the ISO’s GRUB menu: OVMF loads the UKI from fw_cfg as a UEFI application, so the systemd-stub boots the kernel exactly as it will from the key stick’s ESP. COBALT_VM_CMDLINE_EXTRA adds console=ttyS0 through the SMBIOS string io.systemd.stub.kernel-cmdline-extra (systemd-stub(7); honoured only while Secure Boot is off). The ISO stays attached — it is the live medium the initrd mounts by label. The UKI output of nix build .#images.installer-uki is a directory; point the variable at the .efi file inside it.
  • COBALT_VM_OFFLINE=1 isolates the guest (restrict=on): a completed install then proves the embedded closure is sufficient.
  • The installer logs nixos in on the serial line as on the virtual consoles (install profile, services.getty.autologinUser); sudo needs no password there.
  • For the first boot (§6) the installed system has no serial console. The systemd-boot menu is visible on the serial line through the firmware’s console: press e; the cursor is at the START of the line, so type console=ttyS0,115200n8 with a trailing space (or press End first), then Enter — text typed without the space glues onto init= and stage 1 ends in emergency mode (“Failed to start Find NixOS closure”). lanzaboote’s stub accepts an edited command line only while Secure Boot is off (rust/uefi/stub/src/common.rs).
  • The operator’s login shell starts zellij (as on macOS, home/modules/cli.nix), and fish’s vi mode prefixes output with cursor-shape escapes (ESC[1 q). A script typing into the serial line should quit zellij first (Ctrl-q, or zellij kill-all-sessions -y once it is up) and strip CSI sequences before matching output.

6. First boot

Section titled “6. First boot”
Terminal window
./scripts/vm-rehearsal.sh boot

There is no console=ttyS0 in the installed system’s kernel parameters, so the boot menu, stage 1 and the login run in the graphics window; the terminal only carries the QEMU monitor (Ctrl-a c, Ctrl-a x quits).

Expected sequence: the systemd-boot menu lists the NixOS generation and cobalt installer (recovery) (the latter is never the default) → stage 1 finds the header partition on the USB disk → the unlock: with the emulated token it completes without any PIN or touch interaction; without it, the passphrase prompt appears (the install left fido2-device=auto in crypttab with no FIDO2 keyslot — whether systemd falls back to the passphrase promptly or waits for a token is one of the things this boot verifies) → LVM activates, rollback-root resets @ from @root-blank → login (only with the passwords from §1).

Checks (sudo where needed):

  • bootctl status — plain OVMF: Secure Boot unsupported/disabled; OVMFFull: Secure Boot: disabled (setup).
  • cryptsetup luksDump /dev/disk/by-partlabel/disk-a-usb-reserved — a passphrase slot, a systemd-recovery token, and a systemd-fido2 token only if the emulated key was enrolled.
  • lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS and findmnt / /nix /home /persist /boot — the layout of hosts/cobalt/disko.nix, /boot on the USB disk’s ESP.
  • sudo btrfs subvolume list / — @, @home, @nix, @persist, @root-blank.
  • Impermanence: sudo touch /root/marker, reboot, confirm it is gone; /persist/var/lib/sops-nix/key.txt and /var/lib/sbctl (bind-mounted) survive.
  • systemctl --failed, journalctl -b -p warning.
  • usbguard list-devices — the QEMU disk, tablet and (if present) the CanoKey are present at boot and therefore kept (presentDevicePolicy = "keep"); hot-plug a second stick from the monitor (Ctrl-a c, then drive_add 0 id=stick,if=none,file=/path/extra.img,format=raw and device_add usb-storage,id=stick,drive=stick,bus=xhci.0, as in nixos/tests/usbguard.nix) and confirm it is blocked.
  • Recovery entry: reboot, choose cobalt installer (recovery): the live installer must come up from the key’s own COBALT_INSTALL partition — with the CD-ROM absent, this proves that the key stick boots its own installer.
  • Secure Boot (OVMFFull only): sudo sbctl enroll-keys --microsoft && sudo sbctl verify, reboot, bootctl status → Secure Boot: enabled (user); the recovery UKI must still boot (it is signed), and with Secure Boot on the stub ignores an edited command line.
  • sudo nixos-rebuild switch --flake path:/persist/etc/cobalt/config#cobalt — must be a no-op rebuild that needs nothing from the network.

7. Repeating

Section titled “7. Repeating”

prepare erases everything and starts over. To re-run only the boot phase, copy vm/usb.qcow2, vm/nvme.qcow2, vm/efivars.fd (and vm/canokey-state) right after the install and restore the copies instead of reinstalling. COBALT_VM_DRY_RUN=1 prints the QEMU command line without starting anything.

What the rehearsal validates — and what it cannot

Section titled “What the rehearsal validates — and what it cannot”

Validated end to end:

  • install.sh logic: the device-id checks, the refusal to run from the key, the by-id confirmations.
  • The disko layout, including the order (USB first, then NVMe), the detached LUKS2 header on a second device, the pinned cryptsetup parameters, LVM-on-LUKS, the btrfs subvolumes and the @root-blank snapshot.
  • Offline nixos-install from the embedded closure, /persist seeding, sops key placement and declarative passwords, lanzaboote’s boot entries, the recovery UKI and the iso9660 copy on the key.
  • systemd stage 1 with the header on a USB device (DefaultDeviceTimeoutSec, the USB/NVMe initrd modules), recovery-key and passphrase unlock, the per-boot root rollback, the /persist allowlist, USBGuard’s present-device policy, the desktop coming up.
  • With OVMFFull: enrolling the sbctl keys from Setup Mode and booting with Secure Boot enabled.
  • With the canokey device: systemd-cryptenroll --fido2-device=auto against the detached header and the FIDO2 unlock path in the initrd.

Not validated — do not read a clean VM run as proof of these:

  • No real YubiKey. The emulated token answers without PIN entry or touch, so the PIN prompt timing, the “token must be inserted before power-on” behaviour (systemd issue #43342) and YubiKey-specific quirks remain untested. Without the device, FIDO2 is skipped altogether (COBALT_SKIP_FIDO2=1).
  • Firmware. OVMF is not Insyde: no supervisor password, a different Setup Mode menu, different boot-order handling, no option ROMs — the --microsoft enrollment is tested only against OVMF’s expectations. Secure Boot enrollment in the VM depends on the Setup-Mode variable template; on the laptop it depends on the firmware’s “Reset to Setup Mode” working.
  • USB enumeration timing. QEMU’s stick is instantly present and uses bulk-only transport; a real stick may enumerate late, use UAS, or re-enumerate under load — the 180 s device timeout and the “device did not appear” failure mode are only exercised in the easy case.
  • Device names. The by-id names are QEMU’s; my/private.nix must be re-pointed at the real serials before the hardware install.
  • Hardware not emulated: the MT7902 Wi-Fi/Bluetooth and its firmware, the Radeon 610M (amdgpu, Plasma runs on a plain VGA device), AMD microcode, amd_pstate/power-profiles-daemon, the SN740’s real sector geometry and APST behaviour, the IOMMU (amd_iommu=on iommu=force are inert in the VM), the touchpad, suspend/resume.
  • Performance and timing under TCG say nothing about the laptop.
  • Interop: the exFAT interop partition with macOS/Windows and the VeraCrypt container are not part of this flow.