Runbook — first installation

Status: written against the evaluated configuration; not yet exercised on hardware. Rehearse in a VM (two virtual disks: a small one for the “USB key”, a larger one for the “NVMe”) before touching the laptop.

What you need

• The laptop (Acer A315-24P), its NVMe empty or expendable.
• Two USB sticks: the key stick (≥ 32 GB, becomes /boot + LUKS header + installer + interop) and a temporary stick (≥ 4 GB) for the first boot. disko formats the key stick during installation, so the installer cannot run from it the first time.
• A YubiKey with a FIDO2 PIN set (ykman fido access change-pin).
• Another medium for the LUKS header backup and the recovery key (paper, or a stick you keep elsewhere).
• Your workstation with Nix (the Mac): builds the image, prepares secrets.

1. Prepare secrets (workstation)

1. Host age key: age-keygen -o host-age.txt → add its public key to .sops.yaml as &cobalt.
2. secrets/users.yaml (password hashes) and secrets/mihomo.yaml (proxy profile): see Runbook — secrets.
3. sops updatekeys secrets/*.yaml.
4. Fill my/private.nix (identity, disks.usb / disks.nvme by-id paths). If you do not know the by-id paths yet, boot the temporary stick once, read ls -l /dev/disk/by-id/, then come back.

2. Build and flash the installer (workstation)

nix build path:.#images.installer      # path: so my/private.nix is included → embedded closure == target
./scripts/flash-usb.sh result/iso/cobalt-installer.iso disk4     # macOS: diskN ; Linux: /dev/sdX

The image also carries the installer’s Unified Kernel Image at /boot/cobalt-installer.efi (unsigned; nix build path:.#images.installer-uki builds it alone). The script signs it onto the key stick in step 7.

Copy host-age.txt onto another small medium (a plain-FAT stick, or the temporary stick’s spare space): the installer asks for it and it must not travel inside the image. my/private.nix and secrets/ are already in the image (path: reference).

3. Install (laptop, booted from the temporary stick)

sudo COBALT_CONFIG=/etc/cobalt/config /etc/cobalt/install.sh

The script asks for: the host age key, confirmation of both device ids (typed exactly), the LUKS passphrase (the recovery fallback — long diceware), a YubiKey touch (FIDO2 enrollment), where to write the header backup. It runs the disko script that was generated for this configuration at build time once, seeds /persist (age key, Secure Boot keys), copies the embedded system into the target store (nixos-install --system: no evaluation, no builds, no network), then (step 7) writes the live image itself — the iso9660 volume COBALT_INSTALL — into the key stick’s installer partition and places the installer’s signed Unified Kernel Image on the ESP as EFI/Linux/cobalt-installer.efi (signed with the same db key as the system).

If the image was built with placeholder disk ids or without secrets/users.yaml, the script stops and says so. COBALT_ONLINE=1 is the escape hatch: it asks for the missing data, re-evaluates the configuration with --flake and rebuilds what nixpkgs does not allow to be substituted — hundreds of small derivations whose build inputs come from the network, so it is neither offline nor quick.

VM rehearsal without a hardware token: COBALT_SKIP_FIDO2=1 skips the FIDO2 enrollment (the recovery key is still generated and the script prints a notice); the passphrase keyslot then unlocks at boot. Enroll the token later from the installed system: sudo systemd-cryptenroll --fido2-device=auto /dev/disk/by-partlabel/disk-a-usb-reserved.

4. First boot

1. Firmware: set a supervisor password; put Secure Boot into Setup Mode; keep the BIOS updated.
2. Insert the key stick and the YubiKey before powering on (the PIN prompt comes before token detection).
3. Boot → FIDO2 PIN + touch → Plasma login.
4. sudo sbctl enroll-keys --microsoft && sudo sbctl verify → reboot → enable Secure Boot → bootctl status shows Secure Boot: enabled (user).
5. Checks: cryptsetup luksDump /dev/disk/by-partlabel/disk-a-usb-reserved (FIDO2 + recovery slots), usbguard list-devices, nmcli device, systemctl status mihomo, flatpak list.
6. Boot menu: bootctl list shows the NixOS generations plus cobalt installer (recovery) (type #2 entry, /boot/EFI/Linux/cobalt-installer.efi); sudo sbctl verify /boot/EFI/Linux/cobalt-installer.efi reports it signed. lanzaboote’s loader.conf keeps default nixos-*, so the newest generation stays the default.

If stage 1 fails after the unlock (emergency mode): the emergency shell cannot be opened — root is locked and boot.initrd.systemd.emergencyAccess stays off on purpose (an unauthenticated shell in the initrd would sit next to the unlocked /sysroot). Boot the recovery entry (the installer UKI on the key stick) and inspect from there.

5. Afterwards

• Clone the key stick (dd whole-device to a second stick) and keep it elsewhere.
• Rebuild from the installed config: sudo nixos-rebuild switch --flake path:/persist/etc/cobalt/config#cobalt.
• Never run disko --mode format again on this layout.

Recovery

The installer on the key stick

Select cobalt installer (recovery) in the boot menu (key stick inserted; no other medium labelled COBALT_INSTALL attached, or stage 1 may pick that one). It boots the same live system as the temporary stick, from the iso9660 image in the key’s installer partition, with Secure Boot left enabled: the entry is one signed file (kernel + initrd + command line + os-release), checked by the firmware against the enrolled db key. It is a rescue environment — install.sh refuses to run from it (step 1: disko would destroy the medium it runs from); a full reinstall still needs a temporary stick.

Verified from source (nixpkgs 55bfb4d, lanzaboote v1.2.0, systemd 261) versus still unverified on hardware:

• Verified: lzbt install only deletes under EFI/nixos/ (everything it does not know) and EFI/Linux/ names starting with nixos- (rust/tool/systemd/src/install.rs 138–153; test keep_unrelated_files_on_esp). It rewrites loader/loader.conf and EFI/{BOOT,systemd}/*.efi and never touches loader/entries/ or other EFI/<dir>/. EFI/Linux/cobalt-installer.efi survives rebuilds.
• Verified: the UKI’s command line is the ISO’s own (init=…/init root=fstab loglevel=4 lsm=…); the medium is located through the initrd fstab (/dev/disk/by-label/COBALT_INSTALL, iso9660) — hence the image copy, not a file copy (the installer initrd has no ext4). nix eval .#images.installer-uki.drvPath passes.
• Verified: systemd-boot picks the default entry by the default nixos-* glob over entry ids, never by sort order alone (src/boot/boot.c, config_find_entry).
• Unverified on hardware: firmware acceptance of the sbctl-signed UKI, the menu entry’s appearance, by-label resolution with only the key inserted, and the image fitting the 16 GB partition (the script skips with a message if it does not). Nothing has been booted yet.

Opening the NVMe from another medium

Boot any NixOS live medium with the key stick inserted:

cryptsetup open --header /dev/disk/by-partlabel/disk-a-usb-reserved /dev/disk/by-id/<the NVMe, disks.nvme in my/private.nix> cryptroot
vgchange -ay
mount -o subvol=@ /dev/vg/root /mnt && mount -o subvol=@nix /dev/vg/root /mnt/nix \
  && mount -o subvol=@persist /dev/vg/root /mnt/persist && mount /dev/disk/by-partlabel/disk-a-usb-ESP /mnt/boot
nixos-enter --root /mnt

Without the key stick (or its header backup) the NVMe cannot be opened — by design.