docs: vm-install skill, TESTPLAN section B and TODO for the boot loader test and the full automated suite

.claude/skills/vm-install/SKILL.md

@@ -70,12 +70,93 @@ and wait with an until-loop on the log, e.g.
   on); `fatal library error, lookup self` in the chroot is harmless.
 - QMP screenshots don't work (virgl, "no surface"): watch the window, the
   serial log, or the install log.
+- `pkill -f`/`pgrep -f` with a pattern that also appears in your own command
+  line kills or matches your own shell (exit 144). Use `[q]emu` bracket
+  patterns, or kill by PID from `ps -eo pid,args`, in a separate command.
+- A wait loop that runs `pgrep -f "x"` over ssh matches its own ssh command:
+  write `"[x]"`. And `systemctl --wait start` on a unit that already
+  finished waits for ever; poll `systemctl is-active` instead.
+- The live script must not wait for `systemctl is-system-running --wait`: it
+  is itself the running `kernel-command-line.service` job (deadlock). It waits
+  for `pacman-init.service` to be active instead (else "no secret key
+  available to sign with").
+- Only one VM at a time (port 2222); a second `vminstall.sh` refuses while
+  qemu runs. Run several installs one after the other from a script, and
+  give `--force` runs a terminal or delete `disk.qcow2` first (the YES
+  prompt reads /dev/tty).
+- The guest's login shell is fish: send bash through stdin
+  (`ssh ... 'bash -s' < script.sh`), not loops on the command line.
+- `/boot` is root-only: use `sudo` for `ls`/`cat` there, or checks report
+  "no entries" for nothing.
+- `ssh` "timed out during banner exchange" only means QEMU accepted the
+  forwarded port: the guest's sshd isn't answering (still booting, waiting in
+  a boot menu, or a firewall). It says nothing more.
+- Never run `systemctl reboot --firmware-setup --dry-run` to "check": it
+  still sets the EFI flag and the next boot lands in the UEFI setup menu.
+  `bootctl status` shows `Boot into FW: supported` normally, `active` = set.
 
-## The Steamify CachyOS ISO (later)
+## The Steamify CachyOS ISO
 
-`--iso <steamify iso>` installs it the same way if it's archiso-based with
-`cachyos-installer`. The headless installer doesn't run Calamares, so the
-ISO's Steamify step (`steamify-install`) doesn't run by itself: run it in the
-live script after the install (`/usr/local/bin/steamify-install /mnt
-$VM_USER <options>`) when testing the ISO, and check
-`/var/log/steamify-install.log` in the installed system.
+`--iso <steamify iso>` (archiso with `cachyos-installer`). The headless
+installer doesn't run Calamares, so `vminstall-live.sh` runs the ISO's
+`steamify-install /mnt $VM_USER "$VM_STEAMIFY"` itself after the install
+(empty `VM_STEAMIFY` = the Steamify page's default, everything on; ids to
+narrow; `skip` = plain CachyOS) and fails the install if its log doesn't end
+`exit: 0`. The log ends up in `/var/log/steamify-install.log`. The post script
+keeps Steamify's gaming-mode login (SDDM,
+`/etc/sddm.conf.d/10-gamescope-autologin.conf`) instead of forcing the Plasma
+autologin, and the first-boot check then wants the display manager, not
+plasmashell (gamescope doesn't render in the VM: a black window is normal).
+
+## Other options
+
+- `VM_BOOTLOADER=limine|systemd-boot|grub` (default limine) goes into the
+  installer's settings. Use one `VM_DIR` per loader (`~/vms/bl-grub`, ...).
+  cachyos-installer leaves systemd-boot with `#timeout 3` and no default (it
+  waits in the menu for ever): the post script writes `timeout 3` and
+  `default linux-cachyos.conf`, like it sets Limine's `default_entry`.
+- The installer enables **ufw**, which drops the host's ssh
+  (`[UFW BLOCK] DPT=22` in the kernel log): the post script runs
+  `ufw allow 22/tcp`.
+- `VM_CACHE=<dir>` (default `~/vms/pkg-cache`, empty = off) is a host package
+  cache shared as 9p tag `cache`, bound over pacman's cache in the live system
+  and in the new one (Steamify's packages): the first install fills it.
+
+## Testing a boot loader: `scripts/vmtest.sh`
+
+Run this instead of doing the steps below by hand: `scripts/vmtest.sh [--install] [--window] [loader...]`
+(default: every loader the ISO advertises, read from `calamares-online.sh`; it fails when an advertised
+loader has no test). One VM per loader in `~/vms/bl-<loader>` (`--install` builds them from the newest
+Steamify ISO, ~8 min each, ~3 min per test), headless with a Konsole on the logs (`--window` shows the VMs),
+one summary line per loader, `~/vms/bl-<loader>.test.log`, exit status = failed checks. Checks live in
+`share/bootloader-test/*.sh` (one PASS/FAIL line each); the driver is `scripts/vmbootloadertest.sh`.
+Last result: Limine 41, systemd-boot 40, GRUB 40 checks, 0 failed.
+What only shows up per loader: Limine keeps its images under `/boot/<machine-id>/<kernel>/` and copies one
+only when its content changed (same version = untouched), and `remember_last_entry: yes` overrides
+`default_entry` (the test turns it off and restores it); GRUB's other kernel is picked with
+`GRUB_DEFAULT="1>N"` + `grub-mkconfig` (key presses miss its 5 s menu); systemd-boot with
+`bootctl set-oneshot <entry>.conf`. The "kernel update" reinstalls what the installed system's database lists,
+which can be older than what the installer got (mirror skew: 7.2.8 installed, 7.2.7 in the database): a
+downgrade, but still a version change through the same hooks.
+
+Manual version of the same, for one loader:
+
+Steamify's kernel-side parts (`steamify-fremont-poweroff`, `steamify-cros-ec-cec`,
+`leds-valve-dkms`) are DKMS modules for every installed kernel, only installed
+when the VM reports Fremont: start the VM with `./run.sh --fremont`, then run
+`steamify.sh --defaults --options gaming,theme,glyphs,single,launcher,notify,vram,cec,machine,poweroff`
+from the shared repo (`sudo mount -t 9p -o trans=virtio,version=9p2000.L repo /mnt`).
+Then per loader: `dkms status` (3 modules x each kernel), reinstall both kernels
++ headers (`pacman -S linux-cachyos linux-cachyos-lts` + headers: rebuilds
+DKMS, initramfs, and for GRUB the config), reboot into the default kernel and
+into the other one (systemd-boot: `bootctl set-oneshot <entry>.conf`; GRUB or
+Limine: send keys with `scripts/qmpkey.py <vm>/qmp.sock down ret ...`, or
+change `default_entry`), and check `uname -r`, `lsmod`, and `dmesg | grep steamify`.
+The legacy `drm.edid_firmware` removal (`hdmi_remove_boot_param`) is the code
+that differs per loader: seed the parameter in `/etc/default/grub`,
+`/etc/sdboot-manage.conf` or `/etc/default/limine`, source `lib/common.sh`,
+`state.sh`, `hdmi-refresh.sh` from `/mnt`, and run it. To see why a VM won't
+boot without a console, stop it, `qemu-img dd` the first 2 GB to a raw file,
+`dd skip=1M` to cut the ESP, `udisksctl loop-setup` + `mount` it (no root),
+and read `loader/loader.conf` and the entries; or boot the kernel directly with
+`VM_KERNEL/VM_INITRD/VM_APPEND=... console=ttyS0` and `VM_SERIAL=` for a log.

TESTPLAN.md

@@ -95,6 +95,22 @@ change only shows right after Enter).
 - Gamescope itself: gaming mode, Switch to Desktop / Return to Gaming Mode.
 - LED bar light, real TV over HDMI-CEC (remote, TV on/off, sleep/wake).
 
+## B. Boot loaders (`scripts/vmtest.sh`, unattended)
+
+Every loader the ISO advertises (Limine, systemd-boot, GRUB), each on a VM installed from the Steamify ISO
+and started with `--fremont`. Automated: `scripts/vmtest.sh` (see the vm-install skill).
+
+| # | Step | Expect |
+|---|---|---|
+| B1 | first boot after the unattended install | boots by itself (systemd-boot: `timeout`/`default` set by the test setup), SSH answers |
+| B2 | Steamify's state, as its own `*_status` functions | gaming, theme, glyphs, single, launcher, notify, poweroff, cec all on |
+| B3 | OS name | `os-release` NAME/PRETTY_NAME are CachyOS's, `lsb-release` has no "with Steamify"; Limine has `TARGET_OS_NAME` |
+| B4 | loader identity, firmware setup | the expected loader; `Boot into FW: supported` (BIOS item) |
+| B5 | DKMS modules (`leds-valve`, `cros-ec-cec`, `steamify-fremont-poweroff`) | installed for every kernel, files present, `modules-load.d` entry |
+| B6 | legacy `drm.edid_firmware` removal (`hdmi_remove_boot_param`) | detected, removed, initramfs and entries rebuilt, no edid left |
+| B7 | kernel update (both kernels + headers) | DKMS rebuilds 3 modules per kernel, initramfs rebuilt, loader config regenerated |
+| B8 | reboot into the default and the other kernel | both boot, no failed units, power-off fix and LED modules loaded on each |
+
 ## Known gaps
 
 - "Add as non-Steam game" can't be set up at install time (no Steam account
@@ -117,4 +133,5 @@ change only shows right after Enter).
 | 2026-09-28 | same + CEC socket fix, UI label fix | R1.7, R1.10, U1-U8, H1, H4 (desktop), H5 | R1.7 first failed: `cec-audio-control.socket` never enabled (old bug, fixed: `cec_enable` enables it, `FEATURE_VERSION[cec]=2.7.0`); after: 53 PASS / 0 FAIL / 2 SKIP. UI: progress row said "Boot into Boot into" (fixed). Rest pass. Not run: R1.8, R1.9, R2.2, U9, H3 |
 | 2026-09-28 | `release/2.7.0` (VM from vminstall.sh, `~/vms/steamify-vm`) | R1.2 (fresh, CEC socket), R1.8, R1.9, R1.5 via U9, U9, R2.1, R2.2, H3 | all pass except one old bug: theme on (single on) → single off → theme off left single's launcher keys; fixed (`58e43d4`, PR #34) and retested. BIOS dry run F7F0107 → F7F0108 passed; greeter and SDDM reboot checks passed |
 | 2026-09-28 | `release/2.7.0` | H-Real | counted as passed: power-off fix, VRAM booster, LEDs, gamescope and Steam Machine support are unchanged since 2.6.0, which passed on the real Steam Machine. HDMI-CEC's change (enabling `cec-audio-control.socket`) passed with the fake TV in the VM; check the TV remote's volume keys on the real TV after updating (HDMI-CEC shows as update) |
+| 2026-09-29 | `feature/vminstall-bootloader` (Steamify ISO 2026.09.28, `--fremont`) | B1-B8 | Limine 41 / systemd-boot 40 / GRUB 40 checks pass. Found on the way (test setup, not Steamify): ufw blocks ssh, systemd-boot has no timeout/default after cachyos-installer |
 | 2026-09-28 | `release/2.8.0` (`feature/defaults-list`) | G1-G3, R1.1, R1.2 | all pass (VM `--fremont`, fake Steam Machine). Bundle shellcheck clean |

TODO.md

@@ -0,0 +1,55 @@
+# TODO
+
+## GitHub Actions: run the boot loader test in CI
+
+Goal: every push/PR (and on demand) runs the whole automated test (`scripts/vmtest.sh`, see the next
+section: boot loaders plus every suite) on runners, one job per suite or loader, and fails on any `FAIL` line.
+Public repos, so the free Linux runners with KVM apply. Work on `feature/ci-bootloader-test`,
+PR into the release branch. Tick items off as they're done.
+
+- [x] 1. `run.sh`: headless mode (`VM_HEADLESS=1` -> `-display none`, plain virtio-vga); still to do: make `vmtest.sh` and `vmbootloadertest.sh` headless by default (`--window` opts out, a log Konsole opens on a desktop, CI opens none) and copy the repo's run.sh into `$VM_DIR` before starting
+- [ ] 2. `vminstall.sh`: read the ISO's kernel/initramfs without `udisksctl` (bsdtar/7z when there is no session)
+- [ ] 3. `vminstall.sh` / `vmbootloadertest.sh`: no `/dev/tty` prompts and no `pgrep` on the whole host in CI (`CI=1`)
+- [ ] 4. Where the ISO comes from: newest artifact of `steammachine-cachyos-live-iso`'s `build.yml`
+      (`gh run download -R theupriser/steammachine-cachyos-live-iso`), or a `workflow_dispatch` input (run id / URL)
+- [ ] 5. `.github/workflows/bootloader-test.yml`: matrix over `limine, systemd-boot, grub`; steps: install
+      qemu + ovmf, enable /dev/kvm, fetch the ISO, run the script, upload `~/vms/bl-<loader>.test.log` as an artifact
+- [ ] 6. Runner limits: disk (sparse 60 GB qcow2, free space check), RAM (8 GB VM on a 16 GB runner), timeout (60 min)
+- [ ] 7. Cache the pacman packages between runs (`actions/cache` on `~/vms/pkg-cache`, `VM_CACHE`)
+- [ ] 8. Trigger it once on the branch, fix what the runner shows; note the run time in the skill
+- [ ] 8b. Second job, `--fremont` hardware tests from TESTPLAN.md section H (LED driver, CEC with vivid,
+      BIOS dry-run, boot into desktop/gaming and reboot, power-off module): guest-side, so they run on a runner too;
+      needs a way to check without screenshots (gaming mode doesn't render)
+- [ ] 9. `vm-install` skill + `vmtest.sh` header: mention the workflow, and that the advertised-loader check fails CI when a loader has no test
+- [x] 10. TESTPLAN.md: section B for the boot loader test (B1-B8), pointing at `scripts/vmtest.sh`
+
+## Full automated test: everything in TESTPLAN.md that needs no person
+
+One command, `scripts/vmtest.sh [suite...]` (suites: `boot`, `cli`, `menu`, `hw`, `installer`; default all),
+each check prints `PASS`/`FAIL`, the summary counts them, exit status = failures. Each block starts from a
+fresh `ssh-ready` (`scripts/vmreset.sh --fremont`), as TESTPLAN.md says. GitHub runs the same command.
+Guest-side checks live in `share/vmtest/<suite>/`, one file per TESTPLAN block. Adding a row to
+TESTPLAN.md means adding its check here (say which row each check covers in a comment: `# F4`).
+
+Automatable (encode as checks): R1.1-R1.6, R1.7 (`vmcec.sh` already prints PASS/FAIL), R1.8 (toggle matrix),
+R1.9, R1.10, R2.1, G1-G3, F1-F7, H1, H3, H4 (session file), H5, and the boot loader matrix (done).
+Not automatable without a person (stay manual, listed in the summary as SKIP): U1-U9 (screenshots),
+R2.2 (first desktop login look), H-Real (real Steam Machine).
+
+- [ ] S1. Framework: `scripts/vmtest.sh` takes suites, resets the VM per block, tallies PASS/FAIL/SKIP,
+      prints the skipped manual rows; move the loader matrix under suite `boot`
+- [ ] S2. Suite `cli`: G1-G3, F1-F7 (exit codes, `features.state`, boot-desktop unit)
+- [ ] S3. Suite `menu`: R1.1-R1.6, R1.10 with scripted menu input (`vmrun.sh`) and `vmstate.sh` asserts
+- [ ] S4. Suite `hw`: H1, H3 (`WIZARD_BIOS_DRY_RUN=1`), H4, H5, R1.7 (`vmcec.sh --no-sleep`)
+- [ ] S5. Suite `installer`: R2.1 (`vminstallsim.sh`)
+- [ ] S6. R1.8 toggle matrix and R1.9 reboot checks (from the skill), as suite `toggles`
+- [ ] S7. TESTPLAN.md: mark which rows are automated (a column or a tag), keep the results log automatic:
+      the script appends a line to it (date, branch, suites, counts)
+- [ ] S8. Update the `steam-machine-testing` and `cachyos-vm-testing` skills: "run `scripts/vmtest.sh`" first
+
+## Boot loader test (local)
+
+- [x] `scripts/vmtest.sh [--install] [loader...]` runs all advertised loaders, `vmbootloadertest.sh` one
+- [x] guest checks in `share/bootloader-test/` (state, modules, legacy param, kernel update, boot check)
+- [x] first full run of all three through `vmtest.sh`: Limine 41 / systemd-boot 40 / GRUB 40, 0 failed
+- [ ] test the `VM_CACHE` speed-up with a second install