docs: skills and AGENTS.md after 2026-09-29: the ISO release pipeline (new skill), boot entry check and install gotchas, monitor rules, freeing WSL disk, deleting merged branches

This commit is contained in:
theupriser committed 2026-09-29 23:05:29 +02:00
1 parent 195518f0da
commit 0d0279bd20
7 files changed
+160 -3

No files matched your search

+10
View File
@@ -49,6 +49,16 @@ against a VM that gets reinstalled saves the live ISO's host key in `$VM_DIR/kno
install then hangs at its first-boot ssh check ("REMOTE HOST IDENTIFICATION HAS CHANGED"). Never install then hangs at its first-boot ssh check ("REMOTE HOST IDENTIFICATION HAS CHANGED"). Never
ssh to a VM with its `VM_DIR` settings while it installs; fix it with `ssh-keygen -R "[localhost]:<port>" -f $VM_DIR/known_hosts`. ssh to a VM with its `VM_DIR` settings while it installs; fix it with `ssh-keygen -R "[localhost]:<port>" -f $VM_DIR/known_hosts`.
Monitor rules learned on 2026-09-29: **one monitor per job, stop the old one** (`TaskStop`) when a new run
starts, or every run is reported twice; a job that ends by itself (the ISO build on Gitea, `vmtest.sh`) is
best watched with a loop that exits on its terminal states and prints only *changes* (a new step, a finished
job, a FAIL, a new HTTP status), never on a percentage or a PASS count alone. For a Gitea Actions run, poll its
public log (`.../actions/runs/<run>/jobs/<job>/logs`, see `steamify-iso-release`) and grep for the step names.
A chain across systems (Steamify release -> ISO tag -> mirror sync -> Gitea build -> download link) is one
monitor that says each link when it happens and ends on the real check (the download URL answers 200).
When the tool that runs commands stops answering (auto-mode classifier "no verdict"), retry once, then say so
and carry on with reading/local work: nothing running on the PC is affected.
## What the user sees on the PC ## What the user sees on the PC
- **Their terminal**: stream the running job's log into it (the `showlog` unit, `wsl-build-host` - **Their terminal**: stream the running job's log into it (the `showlog` unit, `wsl-build-host`
@@ -5,6 +5,9 @@ description: Use when working on the Steam Machine CachyOS ISO (repo steammachin
# Steam Machine ISO and Steamify's install-time mode # Steam Machine ISO and Steamify's install-time mode
Releasing the ISO (GitHub tag, built and attached on the Gitea mirror, naming, runner and server settings) is
the `steamify-iso-release` skill; this one is for building and testing by hand.
A CachyOS live ISO for the Steam Machine that installs Steamify's default A CachyOS live ISO for the Steam Machine that installs Steamify's default
setup during the install. Two repos: setup during the install. Two repos:
@@ -12,6 +12,12 @@ Rules:
- Never touch unmerged branches; list them and report instead. - Never touch unmerged branches; list them and report instead.
- Never commit or merge to main. - Never commit or merge to main.
Merging a feature/bugfix PR yourself (into a release branch, or `feat/steamify` in the ISO repo): delete the
branch at once, remote and local (`gh pr merge N --merge --delete-branch`, then `git branch -D`, `git fetch
--prune`); the user asked for this on 2026-09-29. Release branches are never deleted (kept as history next to
their tags). Unmerged old branches (`feat/vram-booster`, `refactor/qml-screens`, `backup/*`) are reported, not
touched. The GitHub-side `release delete <tag> --cleanup-tag` is how test ISO releases go.
Steps: Steps:
1. `git fetch --all --tags --prune` 1. `git fetch --all --tags --prune`
2. `git checkout main && git merge --ff-only origin/main` 2. `git checkout main && git merge --ff-only origin/main`
@@ -0,0 +1,101 @@
---
name: steamify-iso-release
description: Use when releasing, debugging or changing how the Steamify ISO is built and published - the GitHub tag that names it, the Gitea mirror (git.upriser.nl) that builds and attaches the 3.2 GB ISO, the Steamify workflows that start it, and every trap met while setting it up (413, privileged runner, sudo dropping variables, mirror syncs deleting tags, upload-artifact@v3, GitHub 429). Also for the Gitea side of steamify-cachyos' own releases.
---
# Steamify ISO release pipeline (set up 2026-09-29, first ISO out the same night)
Tested end to end: release `v2.9.3-dev.2026.09.29-2044` -> `steamify-cachyos-2.9.3-dev.2026.09.29-2044-x86_64.iso`
on git.upriser.nl, GitHub's link to it answers 200.
## The flow (one tag names everything)
1. **GitHub, `iso-release.yml`** (ISO repo `theupriser/steammachine-cachyos-live-iso`, `workflow_dispatch` only):
takes Steamify's newest release (`X.Y.Z`), the time **now in UTC** and makes an **annotated tag**
`vX.Y.Z-[dev.]YYYY.MM.DD-HHMM` plus a GitHub release with the changelog notes and the *direct* download link
on Gitea. `dev.` and a pre-release on `feat/steamify` (test build), none on `master` (a release).
GitHub is the only place that reads a clock.
2. **The mirror syncs** (git.upriser.nl is a pull mirror of GitHub, every 10 minutes): the tag arrives.
3. **Gitea, `steamify-iso.yml`** (`on: push: tags: ['v*']`, skipped on GitHub by `github.server_url`): parses the
tag, builds the ISO on the runner with exactly that Steamify (`STEAMIFY_VERSION`), the tag's time (label) and
the tag without its `v` (`STEAMIFY_ISO_VERSION`: file name, boot menu, `/etc/steammachine-iso-build`), then
attaches ISO + `.sha256` + `.sha1` + `.pkgs.txt` to the mirror's release for that tag
(`akkuman/gitea-release-action`, the run's own token). Never overwrites a release that already has its ISO.
4. **Trigger:** by hand (*Actions -> Steamify ISO release (tag) -> Run workflow*, or
`gh workflow run iso-release.yml -R theupriser/steammachine-cachyos-live-iso --ref feat/steamify`), or by a
new Steamify release: `steamify-cachyos`' `bundle.yml` step "Start the Steamify ISO's release" (secret
`ISO_DISPATCH_TOKEN` there: fine-grained token, only the ISO repo, *Actions: read and write*; without it the
step is skipped). A push does **not** release (a build is 20 minutes and 3.2 GB).
5. Names, for tag `v2.9.3-dev.2026.09.29-2044`: file `steamify-cachyos-2.9.3-dev.2026.09.29-2044-x86_64.iso`,
label `STEAMIFY_2_9_3_20260929_2044` (ISO 9660: 32 chars, `A-Z 0-9 _`). By hand, without a tag: `local`
(`steamify-cachyos-local-x86_64.iso`, `STEAMIFY_<version>_LOCAL`). The build reads no clock.
Total from a Steamify release to the ISO on Gitea: about 30 minutes (release, tag, sync <=10 min, build 15-20).
## Why it is split like this (learned the hard way)
- **A release made only on the mirror disappears** at the next sync: the pull mirror removes tags GitHub doesn't
have, and the release goes with it (one that has files may linger, tagless, and sorts oddly: delete it). A
release on a tag that **comes from GitHub survives** every sync, files included (tested with a 0-byte asset).
Hence: GitHub makes the tag, Gitea only attaches the file. You can't delete a tag on a mirror by hand either.
- GitHub releases take **2 GB per file**; the ISO is 3.2 GB. So GitHub gets notes and a link, Gitea the file.
- **Annotated tags**: a lightweight tag has only its commit's date, so tags on the same commit sort randomly
on Gitea. UTC is fine (the release name says "UTC"; it is 2 h behind Dutch summer time, and nobody cared).
- The GitHub release's link is built from the tag alone (`.../releases/download/<tag>/steamify-cachyos-<tag
without v>-x86_64.iso`), so the Gitea file name must be that: check it with `curl -I -L` (the monitor in the
session did: HTTP 200 = the whole chain is right).
## Gitea side: what the runner and the server need
- **Runner** (act_runner in Docker Compose, network `gitea_gitea`): `config.yaml` -> `container: privileged: true`
(the workflow's `options: --privileged` is **ignored** otherwise; symptom: `mount: .../airootfs/proc: permission
denied`, `failed to setup chroot`; the job log shows `Privileged:false`). ~20 GB free disk.
- **Upload limit: `[repository.release] FILE_MAX_SIZE = 10240`** in `app.ini` (MB; default 2048), restart Gitea.
`[attachment] MAX_SIZE` is for issues, **not** releases: symptom `status: 413` on `.../releases/<id>/assets`.
The release is created *before* the upload, so a failed run leaves an empty release: delete it.
- Logs of a public repo's run are readable without login: `.../actions/runs/<run>/jobs/<job>/logs` (job numbers
count up; re-runs get a new job number). Good for a monitor; no `gh` for Gitea.
- Mirror sync is set to 10 min by the user with their own token; "Synchronize Now" or the API
(`POST /api/v1/repos/<repo>/mirror-sync`) for now. `GITEA_TOKEN` (secret in the ISO repo) would make GitHub
trigger it; not needed.
- **Build traps in the container:** `sudo mkarchiso` in `util-iso.sh` **drops environment variables**:
it needs `sudo --preserve-env=STEAMIFY_ISO_VERSION,STEAMIFY_BUILD_STAMP` (the first Gitea ISO came out as
`-local-`). Step names are evaluated **before** earlier steps set env: don't put `${{ env.X }}` in a name.
Lots of `error: command failed to execute correctly` (pacman hooks wanting systemd) are harmless.
The bash test of the naming must go through `sudo` too, not only call `profiledef.sh` directly.
- Kernel: the ISO is built from the generic `x86_64` repo (must boot on any PC), which lags the `v3/v4/znver4`
repos by a day or so: 7.2.7 on the ISO while installed systems get 7.2.8. Normal, not a bug: the installer
switches to the CPU's optimised repo (znver4 on the 9800X3D) and the first update brings the newer kernel.
## Steamify's own workflow (`steamify-cachyos/.github/workflows/bundle.yml`), one file for both servers
- Build + check run everywhere. shellcheck through `ludeeus/action-shellcheck@2.0.0` (brings its own binary:
act_runner's image has none; GitHub's has). The artifact: `actions/upload-artifact@v4` on GitHub,
`christopherhx/gitea-upload-artifact@v4` elsewhere. **Never write `upload-artifact@v3`**: GitHub fails the
whole workflow for merely mentioning a deprecated version, even in a skipped step.
- Publishing: GitHub release (`gh`, tag + `latest`) only on GitHub; on Gitea the release is `akkuman/gitea-release-action`
with a Gitea-API version check (`RELEASE_TOKEN` secret optional, else the run's token). Steps are guarded by
`github.server_url == 'https://github.com'` (or `!=`). YAML check without PyYAML on the Mac:
`ruby -ryaml -e 'YAML.load_file(...)'`.
- Steamify releases still follow the AGENTS.md flow (`release/X.Y.Z`, `bugfix/`/`feature/` branch, PR into the
release branch, merged and **deleted** by the agent, the release PR into `main` is the user's). Even a CI-only
change is a release (2.9.2, 2.9.3).
## GitHub rate limit that started this (HTTP 429)
`raw.githubusercontent.com` refused Valve's CEC driver source after a day of test installs (each CEC turn-on
downloaded it again; the limit is per address, ~1 per 5 minutes seen). Fixed in Steamify 2.9.1: the driver is
cached in `/var/cache/steamify/cros-ec-cec-<sha256:12>.c` (named after its checksum: a new pin downloads once and
deletes the old file; a damaged file is fetched again). The test VMs share the host's `~/vms/pkg-cache/steamify`
(bound over `/var/cache/steamify` by `vmsuite.sh`, `vmbootloadertest.sh`, `vminstall-live.sh`); hw block 50 tests
the cache offline (`file://`). The source repo `evlaV/linux-integration` is an unofficial mirror without releases:
if it ever vanishes, ship the file with Steamify.
## Checklist for a release
1. Steamify released? (`gh release list -R theupriser/steamify-cachyos`). The ISO gets the newest.
2. `gh workflow run iso-release.yml ... --ref feat/steamify` (test) or `--ref master` (release; master must have
the workflows: it doesn't yet, `feat/steamify` is the ISO repo's working branch).
3. Watch: tag on Gitea (`.../api/v1/repos/theupriser/steamify-cachyos-live-iso/tags`), the run's log (above),
then `curl -I -L` the GitHub link. Old test releases: delete on GitHub (`gh release delete <tag>
--cleanup-tag -y`); the sync removes the tag on Gitea; delete a leftover tagless Gitea release by hand.
+18 -1
View File
@@ -138,6 +138,14 @@ lose the executable bit (`chmod --reference` or call scripts with `bash`); a sui
be a `.host.sh` block; ssh has no Plasma session, the prelude exports one; a one-off network failure can fail be a `.host.sh` block; ssh has no Plasma session, the prelude exports one; a one-off network failure can fail
an install step (LED driver from the AUR), rerun the block before suspecting Steamify. an install step (LED driver from the AUR), rerun the block before suspecting Steamify.
Gotchas found on 2026-09-29: never ssh to a VM with its `VM_DIR` settings while it installs (a leftover loop, also
one started from a laptop command that was stopped, saves the *live ISO's* host key in `$VM_DIR/known_hosts` and
the install then hangs at its first-boot ssh check: `ssh-keygen -R "[localhost]:<port>" -f $VM_DIR/known_hosts`);
`vminstall.sh` now never uses root as the guest user (WSL runs as root: `theupriser`), takes the ISO's loop
device under a lock (parallel installs collided), tolerates the poweroff dropping ssh, waits only for its own
qemu and stops at once when the live system reports `== failed: ...`. A failed install used to look like
"still installing" for 90 minutes: `vmprogress.sh` shows `INSTALL FAILED: <why>`.
Following a run for the user (a table per VM with a bar and a total, only on a change; their terminal and Following a run for the user (a table per VM with a bar and a total, only on a change; their terminal and
the VM's screen): the `progress-report` skill, with `scripts/vmprogress.sh` as the monitor. the VM's screen): the `progress-report` skill, with `scripts/vmprogress.sh` as the monitor.
@@ -149,7 +157,16 @@ loader has no test). One VM per loader in `~/vms/bl-<loader>` (`--install` build
Steamify ISO, ~8 min each, ~3 min per test), headless with a Konsole on the logs (`--window` shows the VMs), Steamify ISO, ~8 min each, ~3 min per test), headless with a Konsole on the logs (`--window` shows the VMs),
one summary line per loader, `~/vms/bl-<loader>.test.log`, exit status = failed checks. Checks live in one summary line per loader, `~/vms/bl-<loader>.test.log`, exit status = failed checks. Checks live in
`share/bootloader-test/*.sh` (one PASS/FAIL line each); the driver is `scripts/vmbootloadertest.sh`. `share/bootloader-test/*.sh` (one PASS/FAIL line each); the driver is `scripts/vmbootloadertest.sh`.
Last result: Limine 41, systemd-boot 40, GRUB 40 checks, 0 failed. Last result (2026-09-29, ISO with Steamify 2.9.1): Limine 43, systemd-boot 42, GRUB 42 checks, 0 failed.
The boot entry check (B1, `boot-check.sh`, after every reboot): the system booted through the loader's **own**
EFI entry (`BootCurrent`, not `auto_created_boot_option`, a real partition GUID not `HD(0,GPT,0000...)`). It
found that the installer never registered systemd-boot (bootctl skips the EFI variables in a chroot): OVMF then
tried PXE/HTTP boot/EFI shell first, 4-5 minutes per boot, or "no bootable device". The ISO's `steamify-install`
now runs `efibootmgr` from the live system (a real partition, first in the order) and mounts the new system's
`/var/log` (`@log` subvolume) first so its logs (`steamify-install.log`, `steamify-bootentry.log`) survive.
To read a VM's boot entries without booting it: copy `vars.fd`, `virt-fw-vars -i <copy> --print` (package
`virt-firmware`). The power-off check accepts "loaded" or "tried at boot: No such device" (journal, not dmesg:
the VM has no AMD GPIO controller, the module refuses on purpose).
What only shows up per loader: Limine keeps its images under `/boot/<machine-id>/<kernel>/` and copies one What only shows up per loader: Limine keeps its images under `/boot/<machine-id>/<kernel>/` and copies one
only when its content changed (same version = untouched), and `remember_last_entry: yes` overrides only when its content changed (same version = untouched), and `remember_last_entry: yes` overrides
`default_entry` (the test turns it off and restores it); GRUB's other kernel is picked with `default_entry` (the test turns it off and restores it); GRUB's other kernel is picked with
+13 -1
View File
@@ -1,6 +1,6 @@
--- ---
name: wsl-build-host name: wsl-build-host
description: Use when building or testing the Steam Machine ISO, or running the VM tests (vmtest.sh), on the user's Windows PC - Arch Linux in WSL2 (`ssh wsl`), with podman, QEMU + KVM and WSLg windows on the Windows desktop. Covers access, syncing working trees from the laptop, the podman ISO build as root, VM windows vs headless, showing output in the user's WSL terminal, and the WSL pitfalls (closing the window kills it, no udisks, no GPU passthrough). description: Use when building or testing the Steam Machine ISO, or running the VM tests (vmtest.sh), on the user's Windows PC - Arch Linux in WSL2 (`ssh wsl`), with podman, QEMU + KVM and WSLg windows on the Windows desktop. Covers access, syncing working trees from the laptop, the podman ISO build as root, VM windows vs headless, showing output in the user's WSL terminal, freeing disk space, and the WSL pitfalls (closing the window kills it, no GPU passthrough).
--- ---
# WSL build host (Arch on WSL2, the Windows PC) # WSL build host (Arch on WSL2, the Windows PC)
@@ -96,6 +96,18 @@ Windows desktop. SSH sessions and units lack the variables, so set them:
run.sh's default virgl (`virtio-vga-gl`, `gl=on`) renders through WSLg's run.sh's default virgl (`virtio-vga-gl`, `gl=on`) renders through WSLg's
d3d12 Mesa on it, but QMP screenshots then give "no surface". d3d12 Mesa on it, but QMP screenshots then give "no surface".
## Freeing disk space
Everything here is re-creatable (VMs, `~/vms/pkg-cache`, `~/projects/iso-cache`, `build/`, the podman image):
`rm -rf` them when the VMs aren't running (`pgrep -f "^[q]emu"`), `podman system prune -a -f`,
`pacman -Scc --noconfirm`, `fstrim -av`. That freed 68 GB inside WSL (81 -> 13 GB), but Windows only gets it
back when the vhdx is sparse: in an admin PowerShell `wsl --shutdown`, `wsl --manage archlinux --set-sparse
true` (seconds), reopen the Arch window, `fstrim -av` again; else `compact vdisk` in diskpart. A rebuild costs
the caches: a full `vmtest.sh --install` (three loader VMs ~12 min in parallel, the suite VM ~12 min from a
CachyOS ISO the user keeps in `/root/`), and an ISO build ~6-8 min. Don't `wsl --shutdown` while a job runs.
Hand-built ISOs are called `steamify-cachyos-local-x86_64.iso`; releases are built on the Gitea mirror
(`steamify-iso-release` skill). Windows starting WSL at logon is the user's own business, not Steamify's.
## Running the tests here ## Running the tests here
`scripts/vmtest.sh --screen` from `~/projects/steamify-cachyos-dev` (as root, `scripts/vmtest.sh --screen` from `~/projects/steamify-cachyos-dev` (as root,
+9 -1
View File
@@ -4,7 +4,7 @@ Test tooling for Steamify (`../steamify-cachyos`) and the Steam Machine ISO (`..
VM scripts (`scripts/`, `run.sh`), the automated test (`scripts/vmtest.sh`, checks in `share/vmtest/` and VM scripts (`scripts/`, `run.sh`), the automated test (`scripts/vmtest.sh`, checks in `share/vmtest/` and
`share/bootloader-test/`), `TESTPLAN.md` (keep it current: rows and a results line after every run) and the `share/bootloader-test/`), `TESTPLAN.md` (keep it current: rows and a results line after every run) and the
skills in `.claude/skills/` (vm-install, cachyos-vm-testing, steam-machine-testing, steam-machine-iso, skills in `.claude/skills/` (vm-install, cachyos-vm-testing, steam-machine-testing, steam-machine-iso,
wsl-build-host, progress-report). wsl-build-host, progress-report, steamify-iso-release, steamify-branch-cleanup).
## Running the tests (do this, don't re-derive it) ## Running the tests (do this, don't re-derive it)
- Everything: `scripts/vmtest.sh --screen` (detached `screen` session `vmtest`), then read only - Everything: `scripts/vmtest.sh --screen` (detached `screen` session `vmtest`), then read only
@@ -36,6 +36,14 @@ wsl-build-host, progress-report).
`scripts/vmtest.sh --screen`, read `~/vms/last-test.txt`, fix what fails. `scripts/vmtest.sh --screen`, read `~/vms/last-test.txt`, fix what fails.
- Open work: `TODO.md` (the CEC driver download, the GitHub Actions workflow, P3). - Open work: `TODO.md` (the CEC driver download, the GitHub Actions workflow, P3).
## Releases (2026-09-29; details in the `steamify-iso-release` skill)
- Steamify: `release/X.Y.Z` -> PR into `main` (the user's) -> GitHub release, and the same workflow on the Gitea
mirror. A new Steamify release starts the ISO's release (`ISO_DISPATCH_TOKEN`); a CEC driver download is cached.
- The ISO: GitHub makes the tag `vX.Y.Z-[dev.]YYYY.MM.DD-HHMM` (UTC) and a release with the notes; the Gitea
mirror (git.upriser.nl, pull mirror, 10 min) builds on that tag and attaches the 3.2 GB ISO, named after the tag.
A release that exists only on the mirror is deleted by its next sync: the tag must come from GitHub.
- Merged feature/bugfix branches are deleted at once (`gh pr merge --delete-branch`). Push every commit.
## Where the tests run ## Where the tests run
The tests run on the user's PC (Ryzen 9800X3D, 64 GB, **WSL2**), started from a laptop over ssh (the user's own skill, or The tests run on the user's PC (Ryzen 9800X3D, 64 GB, **WSL2**), started from a laptop over ssh (the user's own skill, or
`scripts/vmtest-remote.sh <ssh-host> [args]`: pulls the pushed branch there, starts `vmtest.sh --screen`, waits for `scripts/vmtest-remote.sh <ssh-host> [args]`: pulls the pushed branch there, starts `vmtest.sh --screen`, waits for