steamify-cachyos-dev/TODO.md

TODO

GitHub Actions: run the boot loader test in CI

Goal: every push/PR (and on demand) runs the whole automated test (scripts/vmtest.sh, see the next section: boot loaders plus every suite) on runners, one job per suite or loader, and fails on any FAIL line. Public repos, so the free Linux runners with KVM apply. Work on feature/ci-bootloader-test, PR into the release branch. Tick items off as they're done.

• 1. run.sh: headless mode (VM_HEADLESS=1 -> -display none, plain virtio-vga); still to do: make vmtest.sh and vmbootloadertest.sh headless by default (--window opts out, a log Konsole opens on a desktop, CI opens none) and copy the repo's run.sh into $VM_DIR before starting
• 2. vminstall.sh: read the ISO's kernel/initramfs without udisksctl (bsdtar/7z when there is no session)
• 3. vminstall.sh / vmbootloadertest.sh: no /dev/tty prompts and no pgrep on the whole host in CI (CI=1)
• 4. Where the ISO comes from: newest artifact of steammachine-cachyos-live-iso's build.yml (gh run download -R theupriser/steammachine-cachyos-live-iso), or a workflow_dispatch input (run id / URL)
• 5. .github/workflows/bootloader-test.yml: matrix over limine, systemd-boot, grub; steps: install qemu + ovmf, enable /dev/kvm, fetch the ISO, run the script, upload ~/vms/bl-<loader>.test.log as an artifact
• 6. Runner limits: disk (sparse 60 GB qcow2, free space check), RAM (8 GB VM on a 16 GB runner), timeout (60 min)
• 7. Cache the pacman packages between runs (actions/cache on ~/vms/pkg-cache, VM_CACHE)
• 8. Trigger it once on the branch, fix what the runner shows; note the run time in the skill
• 8b. Second job, --fremont hardware tests from TESTPLAN.md section H (LED driver, CEC with vivid, BIOS dry-run, boot into desktop/gaming and reboot, power-off module): guest-side, so they run on a runner too; needs a way to check without screenshots (gaming mode doesn't render)
• 9. vm-install skill + vmtest.sh header: mention the workflow, and that the advertised-loader check fails CI when a loader has no test
• 10. TESTPLAN.md: section B for the boot loader test (B1-B8), pointing at scripts/vmtest.sh

Full automated test: everything in TESTPLAN.md that needs no person

One command, scripts/vmtest.sh [suite...] (suites: boot, cli, menu, hw, installer; default all), each check prints PASS/FAIL, the summary counts them, exit status = failures. Each block starts from a fresh ssh-ready (scripts/vmreset.sh --fremont), as TESTPLAN.md says. GitHub runs the same command. Guest-side checks live in share/vmtest/<suite>/, one file per TESTPLAN block. Adding a row to TESTPLAN.md means adding its check here (say which row each check covers in a comment: # F4).

Automatable (encode as checks): R1.1-R1.6, R1.7 (vmcec.sh already prints PASS/FAIL), R1.8 (toggle matrix), R1.9, R1.10, R2.1, G1-G3, F1-F7, H1, H3, H4 (session file), H5, and the boot loader matrix (done). Not automatable without a person (stay manual, listed in the summary as SKIP; no screenshot testing): U1-U9, R2.2 (first desktop login look), H-Real (real Steam Machine).

• S1. Framework (vmtest.sh runs boot + every suite, --screen, summary in ~/vms/last-test.txt): scripts/vmtest.sh takes suites, resets the VM per block, tallies PASS/FAIL/SKIP, prints the skipped manual rows; move the loader matrix under suite boot
• S2. Suite cli (55 checks pass): G1-G3, F1-F7 (exit codes, features.state, boot-desktop unit)
• S3. Suite menu (36 checks pass): R1.1-R1.6, R1.10 with scripted menu input (vmrun.sh) and vmstate.sh asserts
• S4. Suite hw (70 checks pass incl. H3 BIOS dry-run): H1, H3 (WIZARD_BIOS_DRY_RUN=1), H4, H5, R1.7 (vmcec.sh --no-sleep)
• S5. Suite installer (4 pass): R2.1 (vminstallsim.sh)
• S6. R1.8 toggle matrix as suite toggles (42 pass, 1 expected skip); R1.9 reboot checks still to add
• S7. TESTPLAN.md: mark which rows are automated (a column or a tag), keep the results log automatic: the script appends a line to it (date, branch, suites, counts)
• S8. Update the steam-machine-testing and cachyos-vm-testing skills: "run scripts/vmtest.sh" first

Boot loader test (local)

• scripts/vmtest.sh [--install] [loader...] runs all advertised loaders, vmbootloadertest.sh one
• guest checks in share/bootloader-test/ (state, modules, legacy param, kernel update, boot check)
• first full run of all three through vmtest.sh: Limine 41 / systemd-boot 40 / GRUB 40, 0 failed
• test the VM_CACHE speed-up with a second install