From 508eb68ed54ea41effd726d200bbd26600255f96 Mon Sep 17 00:00:00 2001 From: Rick Peters Date: Tue, 29 Sep 2026 11:23:50 +0200 Subject: [PATCH] docs: vm-install skill, TESTPLAN section B and TODO for the boot loader test and the full automated suite --- .claude/skills/vm-install/SKILL.md | 95 +++++++++++++++++++++++++++--- TESTPLAN.md | 17 ++++++ TODO.md | 55 +++++++++++++++++ 3 files changed, 160 insertions(+), 7 deletions(-) create mode 100644 TODO.md diff --git a/.claude/skills/vm-install/SKILL.md b/.claude/skills/vm-install/SKILL.md index fc8ea68..4494133 100644 --- a/.claude/skills/vm-install/SKILL.md +++ b/.claude/skills/vm-install/SKILL.md @@ -70,12 +70,93 @@ and wait with an until-loop on the log, e.g. on); `fatal library error, lookup self` in the chroot is harmless. - QMP screenshots don't work (virgl, "no surface"): watch the window, the serial log, or the install log. +- `pkill -f`/`pgrep -f` with a pattern that also appears in your own command + line kills or matches your own shell (exit 144). Use `[q]emu` bracket + patterns, or kill by PID from `ps -eo pid,args`, in a separate command. +- A wait loop that runs `pgrep -f "x"` over ssh matches its own ssh command: + write `"[x]"`. And `systemctl --wait start` on a unit that already + finished waits for ever; poll `systemctl is-active` instead. +- The live script must not wait for `systemctl is-system-running --wait`: it + is itself the running `kernel-command-line.service` job (deadlock). It waits + for `pacman-init.service` to be active instead (else "no secret key + available to sign with"). +- Only one VM at a time (port 2222); a second `vminstall.sh` refuses while + qemu runs. Run several installs one after the other from a script, and + give `--force` runs a terminal or delete `disk.qcow2` first (the YES + prompt reads /dev/tty). +- The guest's login shell is fish: send bash through stdin + (`ssh ... 'bash -s' < script.sh`), not loops on the command line. +- `/boot` is root-only: use `sudo` for `ls`/`cat` there, or checks report + "no entries" for nothing. +- `ssh` "timed out during banner exchange" only means QEMU accepted the + forwarded port: the guest's sshd isn't answering (still booting, waiting in + a boot menu, or a firewall). It says nothing more. +- Never run `systemctl reboot --firmware-setup --dry-run` to "check": it + still sets the EFI flag and the next boot lands in the UEFI setup menu. + `bootctl status` shows `Boot into FW: supported` normally, `active` = set. -## The Steamify CachyOS ISO (later) +## The Steamify CachyOS ISO -`--iso ` installs it the same way if it's archiso-based with -`cachyos-installer`. The headless installer doesn't run Calamares, so the -ISO's Steamify step (`steamify-install`) doesn't run by itself: run it in the -live script after the install (`/usr/local/bin/steamify-install /mnt -$VM_USER `) when testing the ISO, and check -`/var/log/steamify-install.log` in the installed system. +`--iso ` (archiso with `cachyos-installer`). The headless +installer doesn't run Calamares, so `vminstall-live.sh` runs the ISO's +`steamify-install /mnt $VM_USER "$VM_STEAMIFY"` itself after the install +(empty `VM_STEAMIFY` = the Steamify page's default, everything on; ids to +narrow; `skip` = plain CachyOS) and fails the install if its log doesn't end +`exit: 0`. The log ends up in `/var/log/steamify-install.log`. The post script +keeps Steamify's gaming-mode login (SDDM, +`/etc/sddm.conf.d/10-gamescope-autologin.conf`) instead of forcing the Plasma +autologin, and the first-boot check then wants the display manager, not +plasmashell (gamescope doesn't render in the VM: a black window is normal). + +## Other options + +- `VM_BOOTLOADER=limine|systemd-boot|grub` (default limine) goes into the + installer's settings. Use one `VM_DIR` per loader (`~/vms/bl-grub`, ...). + cachyos-installer leaves systemd-boot with `#timeout 3` and no default (it + waits in the menu for ever): the post script writes `timeout 3` and + `default linux-cachyos.conf`, like it sets Limine's `default_entry`. +- The installer enables **ufw**, which drops the host's ssh + (`[UFW BLOCK] DPT=22` in the kernel log): the post script runs + `ufw allow 22/tcp`. +- `VM_CACHE=` (default `~/vms/pkg-cache`, empty = off) is a host package + cache shared as 9p tag `cache`, bound over pacman's cache in the live system + and in the new one (Steamify's packages): the first install fills it. + +## Testing a boot loader: `scripts/vmtest.sh` + +Run this instead of doing the steps below by hand: `scripts/vmtest.sh [--install] [--window] [loader...]` +(default: every loader the ISO advertises, read from `calamares-online.sh`; it fails when an advertised +loader has no test). One VM per loader in `~/vms/bl-` (`--install` builds them from the newest +Steamify ISO, ~8 min each, ~3 min per test), headless with a Konsole on the logs (`--window` shows the VMs), +one summary line per loader, `~/vms/bl-.test.log`, exit status = failed checks. Checks live in +`share/bootloader-test/*.sh` (one PASS/FAIL line each); the driver is `scripts/vmbootloadertest.sh`. +Last result: Limine 41, systemd-boot 40, GRUB 40 checks, 0 failed. +What only shows up per loader: Limine keeps its images under `/boot///` and copies one +only when its content changed (same version = untouched), and `remember_last_entry: yes` overrides +`default_entry` (the test turns it off and restores it); GRUB's other kernel is picked with +`GRUB_DEFAULT="1>N"` + `grub-mkconfig` (key presses miss its 5 s menu); systemd-boot with +`bootctl set-oneshot .conf`. The "kernel update" reinstalls what the installed system's database lists, +which can be older than what the installer got (mirror skew: 7.2.8 installed, 7.2.7 in the database): a +downgrade, but still a version change through the same hooks. + +Manual version of the same, for one loader: + +Steamify's kernel-side parts (`steamify-fremont-poweroff`, `steamify-cros-ec-cec`, +`leds-valve-dkms`) are DKMS modules for every installed kernel, only installed +when the VM reports Fremont: start the VM with `./run.sh --fremont`, then run +`steamify.sh --defaults --options gaming,theme,glyphs,single,launcher,notify,vram,cec,machine,poweroff` +from the shared repo (`sudo mount -t 9p -o trans=virtio,version=9p2000.L repo /mnt`). +Then per loader: `dkms status` (3 modules x each kernel), reinstall both kernels ++ headers (`pacman -S linux-cachyos linux-cachyos-lts` + headers: rebuilds +DKMS, initramfs, and for GRUB the config), reboot into the default kernel and +into the other one (systemd-boot: `bootctl set-oneshot .conf`; GRUB or +Limine: send keys with `scripts/qmpkey.py /qmp.sock down ret ...`, or +change `default_entry`), and check `uname -r`, `lsmod`, and `dmesg | grep steamify`. +The legacy `drm.edid_firmware` removal (`hdmi_remove_boot_param`) is the code +that differs per loader: seed the parameter in `/etc/default/grub`, +`/etc/sdboot-manage.conf` or `/etc/default/limine`, source `lib/common.sh`, +`state.sh`, `hdmi-refresh.sh` from `/mnt`, and run it. To see why a VM won't +boot without a console, stop it, `qemu-img dd` the first 2 GB to a raw file, +`dd skip=1M` to cut the ESP, `udisksctl loop-setup` + `mount` it (no root), +and read `loader/loader.conf` and the entries; or boot the kernel directly with +`VM_KERNEL/VM_INITRD/VM_APPEND=... console=ttyS0` and `VM_SERIAL=` for a log. diff --git a/TESTPLAN.md b/TESTPLAN.md index 7328773..d6ee341 100644 --- a/TESTPLAN.md +++ b/TESTPLAN.md @@ -95,6 +95,22 @@ change only shows right after Enter). - Gamescope itself: gaming mode, Switch to Desktop / Return to Gaming Mode. - LED bar light, real TV over HDMI-CEC (remote, TV on/off, sleep/wake). +## B. Boot loaders (`scripts/vmtest.sh`, unattended) + +Every loader the ISO advertises (Limine, systemd-boot, GRUB), each on a VM installed from the Steamify ISO +and started with `--fremont`. Automated: `scripts/vmtest.sh` (see the vm-install skill). + +| # | Step | Expect | +|---|---|---| +| B1 | first boot after the unattended install | boots by itself (systemd-boot: `timeout`/`default` set by the test setup), SSH answers | +| B2 | Steamify's state, as its own `*_status` functions | gaming, theme, glyphs, single, launcher, notify, poweroff, cec all on | +| B3 | OS name | `os-release` NAME/PRETTY_NAME are CachyOS's, `lsb-release` has no "with Steamify"; Limine has `TARGET_OS_NAME` | +| B4 | loader identity, firmware setup | the expected loader; `Boot into FW: supported` (BIOS item) | +| B5 | DKMS modules (`leds-valve`, `cros-ec-cec`, `steamify-fremont-poweroff`) | installed for every kernel, files present, `modules-load.d` entry | +| B6 | legacy `drm.edid_firmware` removal (`hdmi_remove_boot_param`) | detected, removed, initramfs and entries rebuilt, no edid left | +| B7 | kernel update (both kernels + headers) | DKMS rebuilds 3 modules per kernel, initramfs rebuilt, loader config regenerated | +| B8 | reboot into the default and the other kernel | both boot, no failed units, power-off fix and LED modules loaded on each | + ## Known gaps - "Add as non-Steam game" can't be set up at install time (no Steam account @@ -117,4 +133,5 @@ change only shows right after Enter). | 2026-09-28 | same + CEC socket fix, UI label fix | R1.7, R1.10, U1-U8, H1, H4 (desktop), H5 | R1.7 first failed: `cec-audio-control.socket` never enabled (old bug, fixed: `cec_enable` enables it, `FEATURE_VERSION[cec]=2.7.0`); after: 53 PASS / 0 FAIL / 2 SKIP. UI: progress row said "Boot into Boot into" (fixed). Rest pass. Not run: R1.8, R1.9, R2.2, U9, H3 | | 2026-09-28 | `release/2.7.0` (VM from vminstall.sh, `~/vms/steamify-vm`) | R1.2 (fresh, CEC socket), R1.8, R1.9, R1.5 via U9, U9, R2.1, R2.2, H3 | all pass except one old bug: theme on (single on) → single off → theme off left single's launcher keys; fixed (`58e43d4`, PR #34) and retested. BIOS dry run F7F0107 → F7F0108 passed; greeter and SDDM reboot checks passed | | 2026-09-28 | `release/2.7.0` | H-Real | counted as passed: power-off fix, VRAM booster, LEDs, gamescope and Steam Machine support are unchanged since 2.6.0, which passed on the real Steam Machine. HDMI-CEC's change (enabling `cec-audio-control.socket`) passed with the fake TV in the VM; check the TV remote's volume keys on the real TV after updating (HDMI-CEC shows as update) | +| 2026-09-29 | `feature/vminstall-bootloader` (Steamify ISO 2026.09.28, `--fremont`) | B1-B8 | Limine 41 / systemd-boot 40 / GRUB 40 checks pass. Found on the way (test setup, not Steamify): ufw blocks ssh, systemd-boot has no timeout/default after cachyos-installer | | 2026-09-28 | `release/2.8.0` (`feature/defaults-list`) | G1-G3, R1.1, R1.2 | all pass (VM `--fremont`, fake Steam Machine). Bundle shellcheck clean | diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..cd44ae7 --- /dev/null +++ b/TODO.md @@ -0,0 +1,55 @@ +# TODO + +## GitHub Actions: run the boot loader test in CI + +Goal: every push/PR (and on demand) runs the whole automated test (`scripts/vmtest.sh`, see the next +section: boot loaders plus every suite) on runners, one job per suite or loader, and fails on any `FAIL` line. +Public repos, so the free Linux runners with KVM apply. Work on `feature/ci-bootloader-test`, +PR into the release branch. Tick items off as they're done. + +- [x] 1. `run.sh`: headless mode (`VM_HEADLESS=1` -> `-display none`, plain virtio-vga); still to do: make `vmtest.sh` and `vmbootloadertest.sh` headless by default (`--window` opts out, a log Konsole opens on a desktop, CI opens none) and copy the repo's run.sh into `$VM_DIR` before starting +- [ ] 2. `vminstall.sh`: read the ISO's kernel/initramfs without `udisksctl` (bsdtar/7z when there is no session) +- [ ] 3. `vminstall.sh` / `vmbootloadertest.sh`: no `/dev/tty` prompts and no `pgrep` on the whole host in CI (`CI=1`) +- [ ] 4. Where the ISO comes from: newest artifact of `steammachine-cachyos-live-iso`'s `build.yml` + (`gh run download -R theupriser/steammachine-cachyos-live-iso`), or a `workflow_dispatch` input (run id / URL) +- [ ] 5. `.github/workflows/bootloader-test.yml`: matrix over `limine, systemd-boot, grub`; steps: install + qemu + ovmf, enable /dev/kvm, fetch the ISO, run the script, upload `~/vms/bl-.test.log` as an artifact +- [ ] 6. Runner limits: disk (sparse 60 GB qcow2, free space check), RAM (8 GB VM on a 16 GB runner), timeout (60 min) +- [ ] 7. Cache the pacman packages between runs (`actions/cache` on `~/vms/pkg-cache`, `VM_CACHE`) +- [ ] 8. Trigger it once on the branch, fix what the runner shows; note the run time in the skill +- [ ] 8b. Second job, `--fremont` hardware tests from TESTPLAN.md section H (LED driver, CEC with vivid, + BIOS dry-run, boot into desktop/gaming and reboot, power-off module): guest-side, so they run on a runner too; + needs a way to check without screenshots (gaming mode doesn't render) +- [ ] 9. `vm-install` skill + `vmtest.sh` header: mention the workflow, and that the advertised-loader check fails CI when a loader has no test +- [x] 10. TESTPLAN.md: section B for the boot loader test (B1-B8), pointing at `scripts/vmtest.sh` + +## Full automated test: everything in TESTPLAN.md that needs no person + +One command, `scripts/vmtest.sh [suite...]` (suites: `boot`, `cli`, `menu`, `hw`, `installer`; default all), +each check prints `PASS`/`FAIL`, the summary counts them, exit status = failures. Each block starts from a +fresh `ssh-ready` (`scripts/vmreset.sh --fremont`), as TESTPLAN.md says. GitHub runs the same command. +Guest-side checks live in `share/vmtest//`, one file per TESTPLAN block. Adding a row to +TESTPLAN.md means adding its check here (say which row each check covers in a comment: `# F4`). + +Automatable (encode as checks): R1.1-R1.6, R1.7 (`vmcec.sh` already prints PASS/FAIL), R1.8 (toggle matrix), +R1.9, R1.10, R2.1, G1-G3, F1-F7, H1, H3, H4 (session file), H5, and the boot loader matrix (done). +Not automatable without a person (stay manual, listed in the summary as SKIP): U1-U9 (screenshots), +R2.2 (first desktop login look), H-Real (real Steam Machine). + +- [ ] S1. Framework: `scripts/vmtest.sh` takes suites, resets the VM per block, tallies PASS/FAIL/SKIP, + prints the skipped manual rows; move the loader matrix under suite `boot` +- [ ] S2. Suite `cli`: G1-G3, F1-F7 (exit codes, `features.state`, boot-desktop unit) +- [ ] S3. Suite `menu`: R1.1-R1.6, R1.10 with scripted menu input (`vmrun.sh`) and `vmstate.sh` asserts +- [ ] S4. Suite `hw`: H1, H3 (`WIZARD_BIOS_DRY_RUN=1`), H4, H5, R1.7 (`vmcec.sh --no-sleep`) +- [ ] S5. Suite `installer`: R2.1 (`vminstallsim.sh`) +- [ ] S6. R1.8 toggle matrix and R1.9 reboot checks (from the skill), as suite `toggles` +- [ ] S7. TESTPLAN.md: mark which rows are automated (a column or a tag), keep the results log automatic: + the script appends a line to it (date, branch, suites, counts) +- [ ] S8. Update the `steam-machine-testing` and `cachyos-vm-testing` skills: "run `scripts/vmtest.sh`" first + +## Boot loader test (local) + +- [x] `scripts/vmtest.sh [--install] [loader...]` runs all advertised loaders, `vmbootloadertest.sh` one +- [x] guest checks in `share/bootloader-test/` (state, modules, legacy param, kernel update, boot check) +- [x] first full run of all three through `vmtest.sh`: Limine 41 / systemd-boot 40 / GRUB 40, 0 failed +- [ ] test the `VM_CACHE` speed-up with a second install