# TODO

## GitHub Actions: run the boot loader test in CI

Goal: every push/PR (and on demand) runs the whole automated test (`scripts/vmtest.sh`, see the next
section: boot loaders plus every suite) on runners, one job per suite or loader, and fails on any `FAIL` line.
Public repos, so the free Linux runners with KVM apply. Work on `feature/ci-bootloader-test`,
PR into the release branch. Tick items off as they're done.

- [x] 1. `run.sh`: headless mode (`VM_HEADLESS=1` -> `-display none`, plain virtio-vga); still to do: make `vmtest.sh` and `vmbootloadertest.sh` headless by default (`--window` opts out, a log Konsole opens on a desktop, CI opens none) and copy the repo's run.sh into `$VM_DIR` before starting
- [ ] 2. `vminstall.sh`: read the ISO's kernel/initramfs without `udisksctl` (bsdtar/7z when there is no session)
- [ ] 3. `vminstall.sh` / `vmbootloadertest.sh`: no `/dev/tty` prompts and no `pgrep` on the whole host in CI (`CI=1`)
- [ ] 4. Where the ISO comes from: newest artifact of `steammachine-cachyos-live-iso`'s `build.yml`
      (`gh run download -R theupriser/steammachine-cachyos-live-iso`), or a `workflow_dispatch` input (run id / URL)
- [ ] 5. `.github/workflows/bootloader-test.yml`: matrix over `limine, systemd-boot, grub`; steps: install
      qemu + ovmf, enable /dev/kvm, fetch the ISO, run the script, upload `~/vms/bl-<loader>.test.log` as an artifact
- [ ] 6. Runner limits: disk (sparse 60 GB qcow2, free space check), RAM (8 GB VM on a 16 GB runner), timeout (60 min)
- [ ] 7. Cache the pacman packages between runs (`actions/cache` on `~/vms/pkg-cache`, `VM_CACHE`)
- [ ] 8. Trigger it once on the branch, fix what the runner shows; note the run time in the skill
- [ ] 8b. Second job, `--fremont` hardware tests from TESTPLAN.md section H (LED driver, CEC with vivid,
      BIOS dry-run, boot into desktop/gaming and reboot, power-off module): guest-side, so they run on a runner too;
      needs a way to check without screenshots (gaming mode doesn't render)
- [ ] 9. `vm-install` skill + `vmtest.sh` header: mention the workflow, and that the advertised-loader check fails CI when a loader has no test
- [x] 10. TESTPLAN.md: section B for the boot loader test (B1-B8), pointing at `scripts/vmtest.sh`

## Full automated test: everything in TESTPLAN.md that needs no person

One command, `scripts/vmtest.sh [suite...]` (suites: `boot`, `cli`, `menu`, `hw`, `installer`; default all),
each check prints `PASS`/`FAIL`, the summary counts them, exit status = failures. Each block starts from a
fresh `ssh-ready` (`scripts/vmreset.sh --fremont`), as TESTPLAN.md says. GitHub runs the same command.
Guest-side checks live in `share/vmtest/<suite>/`, one file per TESTPLAN block. Adding a row to
TESTPLAN.md means adding its check here (say which row each check covers in a comment: `# F4`).

Automatable (encode as checks): R1.1-R1.6, R1.7 (`vmcec.sh` already prints PASS/FAIL), R1.8 (toggle matrix),
R1.9, R1.10, R2.1, G1-G3, F1-F7, H1, H3, H4 (session file), H5, and the boot loader matrix (done).
Not automatable without a person (stay manual, listed in the summary as SKIP): U1-U9 (screenshots),
R2.2 (first desktop login look), H-Real (real Steam Machine).

- [ ] S1. Framework: `scripts/vmtest.sh` takes suites, resets the VM per block, tallies PASS/FAIL/SKIP,
      prints the skipped manual rows; move the loader matrix under suite `boot`
- [ ] S2. Suite `cli`: G1-G3, F1-F7 (exit codes, `features.state`, boot-desktop unit)
- [ ] S3. Suite `menu`: R1.1-R1.6, R1.10 with scripted menu input (`vmrun.sh`) and `vmstate.sh` asserts
- [ ] S4. Suite `hw`: H1, H3 (`WIZARD_BIOS_DRY_RUN=1`), H4, H5, R1.7 (`vmcec.sh --no-sleep`)
- [ ] S5. Suite `installer`: R2.1 (`vminstallsim.sh`)
- [ ] S6. R1.8 toggle matrix and R1.9 reboot checks (from the skill), as suite `toggles`
- [ ] S7. TESTPLAN.md: mark which rows are automated (a column or a tag), keep the results log automatic:
      the script appends a line to it (date, branch, suites, counts)
- [ ] S8. Update the `steam-machine-testing` and `cachyos-vm-testing` skills: "run `scripts/vmtest.sh`" first

## Boot loader test (local)

- [x] `scripts/vmtest.sh [--install] [loader...]` runs all advertised loaders, `vmbootloadertest.sh` one
- [x] guest checks in `share/bootloader-test/` (state, modules, legacy param, kernel update, boot check)
- [x] first full run of all three through `vmtest.sh`: Limine 41 / systemd-boot 40 / GRUB 40, 0 failed
- [ ] test the `VM_CACHE` speed-up with a second install