Skip to content

Hardening: domain metrics at terminal chokepoints, an honest offline shell, and a checkable definition of done

M8 closes the brief (§12, §13, §15) with three deliverables. Metrics: five lab-domain Prometheus families beside the M1 HTTP pair — lab_afk_runs_total/lab_afk_run_duration_seconds recorded at every terminal-outcome writer, lab_tracker_requests_total from an observer seam on the tracker registry, lab_clone_jobs_total/lab_clones_in_flight from the clone job, and lab_instances_active as a scrape-time collector reading active runs ∩ live tmux, so the gauge can never drift from the truth the reaper acts on. Every label set is a fixed vocabulary; the observer seam's signature is func(binding, op string, ok bool) — no error value exists on it, so token bytes and error text structurally cannot reach the metrics layer. PWA: manifest, icons, and a service worker caching the app shell plus the content-hashed bundles it references, with /api, /agent and the probes never intercepted; a Playwright smoke drives the real embedded-UI binary (first-run setup → dashboard → logout → login) via a self-contained webServer script. Definition of done: docs/definition-of-done.md maps all nine §15 items to the exact tests that verify them, honest about residue (no NixOS VM boot, the AFK "Forgejo" is a stateful fake, restart tests rebuild services rather than kill units); ops.md carries the full config/module/metrics reference, audited live — every §8.5 knob proven effective on a running binary, not just parsed.

The adversarial review earned its keep on all three fronts; thirteen findings, none refuted:

  • A decorator must not break type-assertion seams. Wiring the tracker observer wrapped every resolved tracker — and the agent API rescoped comment identity via tk.(*builtin.Tracker), which the wrapper masks. In production (observer always set) every agent comment would have been silently misattributed to the operator, while every test — fixtures resolve bare builtin trackers — stayed green. The rescope is now an interface (tracker.RunScoper) that the decorator forwards and re-wraps, the handler asserts the interface, and a regression test drives the comment through a real observed registry. The general rule joins the seam docs: callers assert capabilities, never concrete backend types.
  • "Every terminal outcome" must enumerate the writers, not the usual suspects. The reaper and Stop report lab_afk_runs_total; the parked-branch Discard kill is a third terminal-outcome writer and reported nothing, permanently under-counting stopped relative to the runs table. reconcile now carries an injected AFKRunEnded seam (manual-kind runs stay outside the metric's vocabulary). Same shape in the clone job: a force-delete cancelling the job after the clone succeeded still counted ready — cancellation now wins over success in an explicitly-tested classification.
  • An offline shell that precaches only the HTML white-screens offline. The worker cached / and the manifest but not the hashed JS/CSS the shell loads, so the first offline open produced HTML with no app. The shell HTML is now parsed for its /assets/ URLs on install and on every fresh navigation — bundles precached, superseded ones pruned (bounding storage without build-time cache stamping), non-hashed statics network-first, and only real text/html navigations may overwrite the shell key. The Go side matches: a missing asset-like path is a diagnosable 404, never the SPA fallback — HTML answered as a module script is a white screen plus a poisoned cache entry.
  • Unauthenticated endpoints doing per-request work get a concurrency cap. /metrics triggers a runs query and a tmux fork per scrape; promhttp now caps in-flight scrapes at 3 (excess → immediate 503) with a 10s gather timeout.
  • Test artifacts are a leak surface. The Playwright webServer kept its throwaway state — vault master key, DB — inside the repo tree, where any path: flake fetch would copy it into the world-readable nix store. The state dir now lives under mktemp -d, and the web fileset additionally excludes the artifact dirs.
  • The module passes --trusted-proxies whenever set, no longer only under proxyAuth.enable — the server also uses the list without proxy auth (X-Forwarded-Proto trust for Secure-cookie detection behind a TLS-terminating proxy).

Status

Accepted. Ships M8; with it every brief milestone (M1–M8) is implemented. §15 items 5/6/8/9 verified by tests on this tree, 2/3/4/7 verified with the external boundary faked (documented per item), item 1 awaits a real NixOS host — the residue is recorded in docs/definition-of-done.md rather than hand-waved.

Considered options

  • Maintained gauge for lab_instances_active (increment/decrement at start/stop). Rejected: every writer becomes a drift hazard; the scrape-time collector reads the same two cheap sources the reaper trusts, and a snapshot failure yields an absent series (the alert signal) while /metrics stays 200.
  • Observer receiving the error for richer labels. Rejected: an error string is an unbounded, token-bearing channel; (binding, op, ok) is the entire vocabulary by construction.
  • Build-time cache stamping for the service worker (inject the build hash into the cache name, evict on activate). Rejected for now: parsing the shell for its asset set achieves precache + pruning with zero build plumbing; revisit if the worker ever needs versioned migrations.
  • Playwright browsers from npx playwright install. Doesn't exec on NixOS (dynamic linking); the config header documents the nixpkgs playwright-driver.browsers recipe the smoke actually runs with.