steamify-cachyos-dev/TESTPLAN.md

Steamify test plan

What to test before a Steamify release, in the VM (--fremont) and on the real Steam Machine. How to run each step: .claude/skills/cachyos-vm-testing. Keep this file up to date: add a section when a feature is added, and a line to the results log after every test run.

Every VM run starts with the session start of the skill (ssh-ready brought up to date), and each numbered block below starts from a fresh ssh-ready (scripts/vmreset.sh --fremont, REPO= the branch checkout). Runs are visible in a Konsole on the VM's desktop.

R. Regression (every release)

R1. Menu on an existing desktop (user theupriser)

#	Step	Expect
R1.1	'q\n'	first-run ticks: everything but Boot into desktop, BIOS
R1.2	'\ny\nm\nq\nn\n'	every item OK, back in the menu with all on
R1.3	scripts/vmstate.sh	SDDM + autologin, Vapor, 17 LEDs owned by the user, DKMS installed for every kernel, steamos-manager active
R1.4	rerun '\nq\n'	"Everything is already the way you want it"
R1.5	re-apply 'a\ny\nm\nq\nn\n', vmstate.sh	no errors, no DUPLICATES
R1.6	steamify.sh --backend status	valid JSON, current VERSION, states match R1.3
R1.7	close the app first, then sudo modprobe vivid; scripts/vmcec.sh --no-sleep	all PASS (2 SKIP in the VM: wake from TV, sleep). The fake remote's Select arrives as Enter: an open app on "All done" restarts the PC
R1.8	toggles (theme, single user, CEC, Steam Machine support off/on)	full matrix in the skill; cmp.sh clean after theme off
R1.10	set an item older in ~/.local/state/steamify/features.state (e.g. cec=2.1.0), 'q\n', then '\ny\nm\nq\nn\n'	menu shows "HDMI-CEC (update)" ticked; the plan lists it as "update"; after the run the file has the current version
R1.9	reboot checks	single user: SDDM logs straight in; everything off: greeter

U. The app (/mnt/ui/steamify-ui, on the VM's desktop)

Drive it with scripts/qmpkey.py <vm dir>/qmp.sock <keys> (QEMU qcodes: up down left right ret esc ctrl-ret ctrl-q r; wait ~1 s between keys and ~12 s after starting the app) and check every screen with a screenshot (spectacle -b -n -a for the active window; the progress screen of a quick change only shows right after Enter).

#	Step	Expect
U1	start after R1.5	header v<VERSION>, Steam Machine chip, every row with "now on", Kernel / LED bar (17) / HDMI-CEC / Version panel
U2	Up/Down over the rows	detail panel follows: body text and "What it changes" per row
U3	an updated feature (bump FEATURE_VERSION, or features.state older)	UPDATE badge, "1 change"
U4	Boot into: Right / Left	Gaming / Desktop, detail panel DESKTOP, change count +1
U5	Enter on a toggle, Enter again	off, back on, count back
U6	Ctrl+Enter	review lists exactly the changes, summary counts right; Esc goes back
U7	Enter (apply)	progress per step ("Boot into desktop", update badge), details log, "All done, N changes applied", Restart now / Back to the menu
U8	after U7	the changes are on the system (e.g. boot-desktop unit, features.state)
U9	r (re-apply), Ctrl+Q	re-apply review; the app quits

R2. Installer mode (scripts/vminstallsim.sh, user isotest, no session)

#	Step	Expect
R2.1	plain --defaults	every item OK, exit: 0, first-login autostart present
R2.2	first desktop login (steam-machine-iso skill)	Vapor layout, primaryActions=3, autostart gone, app opens

F2.8.0 --defaults --list

#	Step	Expect
G1	steamify.sh --defaults --list	exit 0, valid JSON array, no stderr, no header text mixed in
G2	items	actions (bios), retired (kpin, hdmi) excluded; unavailable here (vram, steamgame) excluded; poweroff has parent machine, boot has parent gaming and kind choice
G3	menu / plain --defaults still work	unaffected (regression)

F. Features per release

F2.7.0 --defaults --options / --boot

#	Step	Expect
F1	--options (no value), bogus, boot, bios, --boot (no value), --boot sideways, --skip x	each exit: 1 with a clear error, nothing changed
F2	--options theme --boot desktop (either order)	exit: 1: desktop needs the conversion
F3	--options gaming,theme,glyphs,single,launcher,notify,vram,cec,machine,poweroff --boot desktop	all on, boot-desktop unit enabled; VRAM left out with a warning in the VM
F4	--options theme,single,machine --boot desktop	single brings the conversion; poweroff, CEC, glyphs, launcher, notify recorded off
F5	--options launcher,vram,poweroff	poweroff brings Steam Machine support; conversion off, plasmalogin unchanged
F6	steamify.sh --boot desktop without the conversion	exit: 1
F7	on an install: --boot desktop, again, --boot gamescope	only Boot into changes; "Already starting in desktop"; menu ticks unchanged

H. Steam Machine hardware

H-VM. With the faked Steam Machine (--fremont)

#	Step	Expect
H1	LED driver	leds_valve loaded, 17 valve-leds* nodes owned by the user (no light to see)
H2	HDMI-CEC	R1.7 (vivid: fake TV and remote)
H3	BIOS update	BIOS_VERSION=F7F0107 scripts/vmreset.sh --fremont, WIZARD_BIOS_DRY_RUN=1: whole flow, never flashes
H4	Boot into desktop, reboot	Plasma straight after boot; Boot into gaming: SDDM autologin into gamescope (black in the VM, see the skill; check the session file)
H5	power-off module	DKMS built for every kernel, module loads, "No such device" (no GPIO wake bit in the VM)

H-Real. Only on the real Steam Machine

• Power-off fix: stays off after shutdown, for every new major kernel.
• VRAM booster (the VM has no dmem VRAM region).
• Gamescope itself: gaming mode, Switch to Desktop / Return to Gaming Mode.
• LED bar light, real TV over HDMI-CEC (remote, TV on/off, sleep/wake).

B. Boot loaders (scripts/vmtest.sh, unattended)

Every loader the ISO advertises (Limine, systemd-boot, GRUB), each on a VM installed from the Steamify ISO and started with --fremont. Automated: scripts/vmtest.sh (see the vm-install skill).

#	Step	Expect
B1	first boot after the unattended install, and every reboot	boots by itself through the loader's own EFI boot entry (BootCurrent, a real partition; checked after every reboot), no network boot first (systemd-boot: the entry comes from the ISO's steamify-install since PR #2; timeout/default set by the test setup), SSH answers
B2	Steamify's state, as its own *_status functions	gaming, theme, glyphs, single, launcher, notify, poweroff, cec all on
B3	OS name	os-release NAME/PRETTY_NAME are CachyOS's, lsb-release has no "with Steamify"; Limine has TARGET_OS_NAME
B4	loader identity, firmware setup	the expected loader; Boot into FW: supported (BIOS item)
B5	DKMS modules (leds-valve, cros-ec-cec, steamify-fremont-poweroff)	installed for every kernel, files present, modules-load.d entry
B6	legacy drm.edid_firmware removal (hdmi_remove_boot_param)	detected, removed, initramfs and entries rebuilt, no edid left
B7	kernel update (both kernels + headers)	DKMS rebuilds 3 modules per kernel, initramfs rebuilt, loader config regenerated
B8	reboot into the default and the other kernel	both boot, no failed units, power-off fix and LED modules loaded on each

Known gaps

• "Add as non-Steam game" can't be set up at install time (no Steam account yet); not covered by any installer test.

• ISO: steamify-install and the packagechooser pages still pass --skip (removed in 2.7.0); to be replaced by the Steamify page.

• /etc/plasmalogin.conf missing before Steamify (a VM from vminstall.sh, not Calamares): turning the conversion off leaves an empty file instead of none (harmless; Calamares installs have one, restored from the backup).

• R2.2: the first login's app is the newest release (run-app), so before a release it shows the old version; start the branch's app by hand (steam-machine-iso skill).

• CEC: the driver source comes from an unofficial mirror of Valve's kernel on GitHub; since 2.9.1 it is downloaded once and cached (/var/cache/steamify; the VMs share the host's), so only a first download can hit GitHub's rate limit (HTTP 429).

Results log

Date	Branch / version	Blocks	Result
2026-09-28	feat/defaults-options 2.7.0 (uncommitted --options, --boot)	R1.1-R1.6, R2.1, F1-F7	all pass (VM --fremont)
2026-09-28	same + CEC socket fix, UI label fix	R1.7, R1.10, U1-U8, H1, H4 (desktop), H5	R1.7 first failed: cec-audio-control.socket never enabled (old bug, fixed: cec_enable enables it, FEATURE_VERSION[cec]=2.7.0); after: 53 PASS / 0 FAIL / 2 SKIP. UI: progress row said "Boot into Boot into" (fixed). Rest pass. Not run: R1.8, R1.9, R2.2, U9, H3
2026-09-28	release/2.7.0 (VM from vminstall.sh, ~/vms/steamify-vm)	R1.2 (fresh, CEC socket), R1.8, R1.9, R1.5 via U9, U9, R2.1, R2.2, H3	all pass except one old bug: theme on (single on) → single off → theme off left single's launcher keys; fixed (58e43d4, PR #34) and retested. BIOS dry run F7F0107 → F7F0108 passed; greeter and SDDM reboot checks passed
2026-09-28	release/2.7.0	H-Real	counted as passed: power-off fix, VRAM booster, LEDs, gamescope and Steam Machine support are unchanged since 2.6.0, which passed on the real Steam Machine. HDMI-CEC's change (enabling cec-audio-control.socket) passed with the fake TV in the VM; check the TV remote's volume keys on the real TV after updating (HDMI-CEC shows as update)
2026-09-29	feature/vminstall-bootloader (Steamify ISO 2026.09.28, --fremont)	B1-B8	Limine 41 / systemd-boot 40 / GRUB 40 checks pass. Found on the way (test setup, not Steamify): ufw blocks ssh, systemd-boot has no timeout/default after cachyos-installer
2026-09-28	release/2.8.0 (feature/defaults-list)	G1-G3, R1.1, R1.2	all pass (VM --fremont, fake Steam Machine). Bundle shellcheck clean
2026-09-29	Steamify 2.9.0 (steamify-cachyos main), ISO feat/steamify with PR #2 (2026.09.29), dev main 5b867f8; on the PC (WSL2)	everything automated: B1-B8 (3 loaders), cli, menu, hw, installer, toggles (scripts/vmtest.sh, parallel, MAX_PARALLEL=3)	328 pass, 0 fail (toggles 42 +1 expected skip, hw 70, cli 55, menu 36, installer 4, Limine 41, systemd-boot 40, GRUB 40) in 18 minutes: the first complete parallel run. Found on the way: systemd-boot had no EFI boot entry (installer's bootctl in a chroot), fixed in the ISO (PR #2); vminstall.sh fixes for parallel installs and a root host user
2026-09-29	ISO feat/steamify + PR #3 (/var/log), dev main with the B1 boot entry check	B1-B8	Limine 43 / GRUB 42 pass (boot entry check passes: Limine, cachyos). systemd-boot (reinstalled): 36 pass, 8 fail, all CEC: the driver download got HTTP 429 from GitHub (rate limit); boot entry check passes (Linux Boot Manager); /var/log/steamify-install.log and steamify-bootentry.log readable after boot
2026-09-29	Steamify release/2.9.1 (CEC driver cache), ISO feat/steamify (PRs #2, #3), dev main	everything automated incl. hw block 50 (CEC cache) and the B1 boot entry check	341 pass, 0 fail in 19 minutes (toggles 42 +1 expected skip, hw 77, cli 55, menu 36, installer 4, Limine 43, systemd-boot 42, GRUB 42)
2026-09-30	Steamify 2.9.6 (steamify-cachyos main d93f9ea), the official ISO steamify-cachyos-2.9.6-260930-x86_64.iso from git.upriser.nl (sha256 4a94f308…aec41), dev main; on the PC, headless (CI=1/VM_HEADLESS=1), MAX_PARALLEL=5	everything automated: B1-B8 (3 loaders, --install from the new ISO), cli, menu, hw, installer, toggles (base VM from the same ISO with VM_STEAMIFY=skip)	341 pass, 0 fail (toggles 42 +1 expected skip, hw 77, cli 55, menu 36, installer 4, Limine 43, systemd-boot 42, GRUB 42). Found on the way (test setup, not Steamify or the ISO): with five VMs at once the loaders' first boot took longer than waitssh waited (5 min), so all three stopped with FAIL no SSH although the VMs were up; waitssh now waits up to 15 minutes and the loaders were run again (install kept)