TODO

GitHub Actions: run the boot loader test in CI

Goal: every push/PR (and on demand) runs the whole automated test (scripts/vmtest.sh, see the next section: boot loaders plus every suite) on runners, one job per suite or loader, and fails on any FAIL line. Public repos, so the free Linux runners with KVM apply. Work on feature/ci-bootloader-test, PR into the release branch. Tick items off as they're done.

1. run.sh: headless mode (VM_HEADLESS=1 -> -display none, plain virtio-vga); still to do: make vmtest.sh and vmbootloadertest.sh headless by default (--window opts out, a log Konsole opens on a desktop, CI opens none) and copy the repo's run.sh into $VM_DIR before starting
2. vminstall.sh: read the ISO's kernel/initramfs without udisksctl (bsdtar/7z when there is no session)
3. vminstall.sh / vmbootloadertest.sh: no /dev/tty prompts and no pgrep on the whole host in CI (CI=1)
4. Where the ISO comes from: newest artifact of steammachine-cachyos-live-iso's build.yml (gh run download -R theupriser/steammachine-cachyos-live-iso), or a workflow_dispatch input (run id / URL)
5. .github/workflows/bootloader-test.yml: matrix over limine, systemd-boot, grub; steps: install qemu + ovmf, enable /dev/kvm, fetch the ISO, run the script, upload ~/vms/bl-<loader>.test.log as an artifact
6. Runner limits: disk (sparse 60 GB qcow2, free space check), RAM (8 GB VM on a 16 GB runner), timeout (60 min)
7. Cache the pacman packages between runs (actions/cache on ~/vms/pkg-cache, VM_CACHE)
8. Trigger it once on the branch, fix what the runner shows; note the run time in the skill
8b. Second job, --fremont hardware tests from TESTPLAN.md section H (LED driver, CEC with vivid, BIOS dry-run, boot into desktop/gaming and reboot, power-off module): guest-side, so they run on a runner too; needs a way to check without screenshots (gaming mode doesn't render)
9. vm-install skill + vmtest.sh header: mention the workflow, and that the advertised-loader check fails CI when a loader has no test
10. TESTPLAN.md: section B for the boot loader test (B1-B8), pointing at scripts/vmtest.sh

Full automated test: everything in TESTPLAN.md that needs no person

One command, scripts/vmtest.sh [suite...] (suites: boot, cli, menu, hw, installer; default all), each check prints PASS/FAIL, the summary counts them, exit status = failures. Each block starts from a fresh ssh-ready (scripts/vmreset.sh --fremont), as TESTPLAN.md says. GitHub runs the same command. Guest-side checks live in share/vmtest/<suite>/, one file per TESTPLAN block. Adding a row to TESTPLAN.md means adding its check here (say which row each check covers in a comment: # F4).

Automatable (encode as checks): R1.1-R1.6, R1.7 (vmcec.sh already prints PASS/FAIL), R1.8 (toggle matrix), R1.9, R1.10, R2.1, G1-G3, F1-F7, H1, H3, H4 (session file), H5, and the boot loader matrix (done). Not automatable without a person (stay manual, listed in the summary as SKIP; no screenshot testing): U1-U9, R2.2 (first desktop login look), H-Real (real Steam Machine).

S1. Framework (vmtest.sh runs boot + every suite, --screen, summary in ~/vms/last-test.txt): scripts/vmtest.sh takes suites, resets the VM per block, tallies PASS/FAIL/SKIP, prints the skipped manual rows; move the loader matrix under suite boot
S2. Suite cli (55 checks pass): G1-G3, F1-F7 (exit codes, features.state, boot-desktop unit)
S3. Suite menu (36 checks pass): R1.1-R1.6, R1.10 with scripted menu input (vmrun.sh) and vmstate.sh asserts
S4. Suite hw (70 checks pass incl. H3 BIOS dry-run): H1, H3 (WIZARD_BIOS_DRY_RUN=1), H4, H5, R1.7 (vmcec.sh --no-sleep)
S5. Suite installer (4 pass): R2.1 (vminstallsim.sh)
S6. R1.8 toggle matrix as suite toggles (42 pass, 1 expected skip); R1.9 reboot checks still to add
S7. TESTPLAN.md: mark which rows are automated (a column or a tag), keep the results log automatic: the script appends a line to it (date, branch, suites, counts)
S8. Update the steam-machine-testing and cachyos-vm-testing skills: "run scripts/vmtest.sh" first

Speed: the full run takes about an hour (target 20-25 min)

Where the time goes: boot loader VMs ~3 min each, every suite block restores a snapshot and reboots (~1 min), and every block that applies Steamify downloads Steam and the rest again from the mirrors.

P1 (proven 2026-09-29: full run in 18 min on the PC). Package cache for the suite VM: VM_CACHE shared as 9p and bound over /var/cache/pacman/pkg in vmreset.sh/the prelude (as vminstall-live.sh does); downloads only once
P2 (base image + overlays, own port per suite, MAX_PARALLEL=3, VM_MEM=4G; first complete parallel run 2026-09-29: 328 pass, 18 min). Parallel: each suite (and each boot loader) on its own copy of the VM (cp --reflink of the disk and vars), own VM_PORT and VM_DIR; vmtest.sh starts them together and waits; watch host CPU (16 cores, 6 vCPUs each)
P3. Fewer resets: merge blocks that only read state (cli 10 + the menu R1.1)

Run the tests on the PC (9800X3D, 64 GB) from a laptop

R1. scripts/vmtest-remote.sh <ssh-host> written; try it once: the PC needs the three repos, qemu + OVMF + screen, KVM (on Windows: WSL2 with nested virtualization), the VMs in ~/vms (vmtest.sh --install builds the boot loader VMs, vminstall.sh --iso <CachyOS ISO> with VM_STEAMIFY=skip the plain one), and the Steamify ISO
R1b (done 2026-09-29: full vmtest.sh run there, 328 pass in 18 min; setup in the wsl-build-host skill). The PC is WSL2 (Windows). To check there: /dev/kvm exists (Windows 11, virtualization on in the BIOS, nested virtualization on); %UserProfile%\.wslconfig memory= is raised (WSL2 defaults to half the RAM = 32 GB; 3-4 VMs at 4-8 GB want ~48 GB) and processors=16; keep ~/vms and the repos on WSL's own ext4, not under /mnt/c; the distro is Arch (root): pacman -S qemu-full edk2-ovmf screen; automated runs are headless with no Konsole (view_start returns early; watch with screen -r vmtest or tail -f ~/vms/test.log), a VM for the user gets a window through WSLg (see the wsl-build-host skill). Still to do: a first full vmtest.sh run there
R1c. vminstall.sh without udisks (WSL): losetup -P + mount when udisksctl is missing and it runs as root
R2. With 64 GB and 8c/16t: try MAX_PARALLEL=4 and 8 GB VMs (VM_MEM=8G)

Found 2026-09-29 (first full run on the PC)

B1 check (share/bootloader-test/boot-check.sh, after every reboot; passes on all three): the loader's own EFI boot entry exists, points to a real partition (not HD(0,GPT,0000...)) and was used (efibootmgr BootCurrent); systemd-boot booted through the fallback path unnoticed
ISO: mount @log at $ROOT/var/log before steamify-install writes its logs (PR #3 merged)
Steamify 2.9.1 (PR #53 into release/2.9.1; hw block 50 tests it): the CEC driver comes from one download (raw.githubusercontent.com, no retry, no fallback); a GitHub rate limit (HTTP 429, seen 2026-09-29 after a day of test installs) leaves CEC off. Ship the pinned source with Steamify instead (steamify-cachyos, via a release branch). Done as a cache in /var/cache/steamify (named after its checksum); the VMs share the host's (~/vms/pkg-cache/steamify). Later maybe: ship the file itself (the source repo is an unofficial mirror)
ISO: systemd-boot's EFI boot entry (efibootmgr from the live system), PR #2 merged
vminstall.sh: a poweroff that drops ssh is no failure, wait only for its own VM, ISO loop device under a lock, never root as the guest user

Boot loader test (local)

scripts/vmtest.sh [--install] [loader...] runs all advertised loaders, vmbootloadertest.sh one
guest checks in share/bootloader-test/ (state, modules, legacy param, kernel update, boot check)
first full run of all three through vmtest.sh: Limine 41 / systemd-boot 40 / GRUB 40, 0 failed
test the VM_CACHE speed-up with a second install

8.1 KiB Raw Blame History