Skip to content

Container-ready by default: services.lab.container.enable defaults on, the agent-tools default refs follow the committed versions.env pin, the deploy pin advances only behind a completed publish, and preflight re-pulls the tags

ADR-0052 shipped the container runner behind a deliberately explicit opt-in: every host mutation gated on services.lab.container.enable = false by default, and toolsImages empty with no default. That posture was right for the tracer bullet and wrong for the steady state, which the first real rollout proved (issue #220): the dev host took the module PR, CI bumped the pin, the deploy went green — and container spawns stayed refused, because nothing in that pipeline ever flips a switch that lives in a different repo's host config. An opt-in that every deployment is expected to take is not an opt-in; it is a deploy-time trap. The follow-up question — which image refs should the default name? — was settled against per-merge rebuilds: images are rebuilt only when their inputs change, and every pin that consumes them advances only behind a completed publish.

The pins, decided:

  • container.enable defaults to true: enabling lab means a container-ready host. The provisioning stays gated on the one switch (never on option non-emptiness — that ADR-0052 rule is untouched); what changes is the switch's resting position. A host that must not run containers sets enable = false and gets the byte-identical pre-container unit — the zero-diff guarantee moves from "the default" to "the opt-out", and the nixos-module check pins it there. Consequence accepted deliberately: virtualisation.podman.enable is now on for every lab host, and an operator force-disabling podman without also disabling container surfaces as an eval conflict, which is the honest shape of that contradiction.

  • Agent-tools images build only when their inputs change; versions.env is the pin, and the module's toolsImages default reads it. The publish leg stays path-gated to containers/** (plus the workflow file), exactly as ADR-0051 built it — a merge that touches only server code rebuilds no images. services.lab.container.toolsImages defaults to <toolsImageRepo>:claude-<CLAUDE_CODE_VERSION> / :codex-<CODEX_VERSION>, the CLI versions read from containers/agent-tools/versions.env at eval (toolsImageRepo is a new option, default git.cloonar.com/cloonar/agent-tools). The committed file that drives the build is the same file that derives the consumer refs, so the default refs move exactly when — and only when — the images they name are rebuilt under a new tag. Per-merge <provider>-<git sha> release tags were built first and rejected (below): rebuilding both images on every merge bought precision the pipeline can get cheaper.

  • The deploy pin advances only behind a completed publish. deploy.yml (the job that bumps the coding-lab rev in the host-config repo) now checks out the pushed rev, derives the default refs from its versions.env, and polls the registry — anonymously, via skopeo — until every ref resolves, with a bounded deadline. A merge that changed the agent-tools inputs triggers their publish on the same push the deploy runs on; the wait serializes the two, so the host pin never names a rev whose default refs are unpublished. On timeout the deploy fails and the pin stays — the honest outcome. For the common merge (no agent-tools change) the tags long exist and the gate passes instantly.

  • Preflight pulls first and falls back to the cache — a moving tag actually reaches hosts. The version tags are moving references: a labctl refresh ships as a same-tag re-push (via workflow_dispatch), and ADR-0052's image exists-first probe would pin every host to whatever digest it pulled first, forever. Preflight now runs podman pull unconditionally per configured ref (an up-to-date tag makes that a cheap manifest check), so each lab restart — including every deploy — picks up a moved tag. When the pull fails but a cached image exists, the host degrades to the cache with a logged warning instead of refusing spawns: stale tools beat none while the registry is unreachable. A pull failure with no cache remains a spawn-refusing preflight failure — and the one failure class cmd/lab retries on an interval, republishing through the existing atomic gate, since time (a still-running publish, a registry blip) is what heals it.

  • "Can the image actually be pulled" is proven at every seam, always anonymously. Publish-side: after pushing, the job logs out and re-pulls the tags into a scratch storage root, comparing digests — Forgejo package visibility follows repo/org settings, so a package quietly flipped private fails the publish loudly instead of failing every host's preflight quietly. Deploy-side: the skopeo wait above is the same anonymous check from a second vantage. Host-side: preflight's unconditional pull. Anonymous is the contract at all three because the module provisions no registry credentials — credentialed pulls are a deliberate non-feature until something needs them.

  • The subid default is unchanged and the collision risk is documented, not detected. Default-on means the 100000:65536 range now lands on hosts that never thought about subids — and 100000 is also the range most tools hand a host's first interactive user. Overlap is functionally harmless for podman but weakens the isolation story (two users' containers sharing host uids), and neither NixOS nor lab can reliably detect it at eval (auto-allocated ranges materialize at activation). The escape hatches stay as ADR-0052 built them: override start/count, or subIdRange = null and bring your own entries; ops.md now flags the check explicitly.

Status

Accepted. Settled via issue #220 (2026-07-24), after the 2026-07-23 dev-new rollout diagnosis showed a pin bump alone can never activate the runner; the build-on-change + pin-behind-publish shape replaced a first per-merge-sha-tag design during review of PR #221. Amends ADR-0052 (the enable switch's default; preflight's pull-first/cached-fallback and retry behavior; the gating rule, mount inventory, argv, and refusal semantics are untouched) and ADR-0051 (the version tags gain a consumer — the module default — and the publish job gains the anonymous pull verification; the /opt/lab injection contract and the digest-as-strict-pin discipline are untouched). ADR-0053's "defaultImage may be null" stands — this ADR adds no dev-image default.

Considered options

  • Keep opt-in and document the host-config step harder. Rejected: the failure mode was not missing documentation but a cross-repo seam — the repo that merges container support cannot flip the switch in the repo that deploys hosts, so "documented opt-in" structurally reproduces issue #220's silent no-op deploy. Defaults are the only channel this repo controls end to end.

  • Per-merge <provider>-<git sha> release tags, module default pinned to self.rev. Built first, rejected in review: it re-runs a full image build+publish on every merge to main to keep a labctl-freshness guarantee that build-on-change plus preflight's re-pull delivers for the cases that matter, at a fraction of the CI and registry cost. It also required a dirty-tree fallback in the module default (a revless source has no sha tag), which the versions.env default simply doesn't need — one shape, clean or dirty.

  • A bot commit writing digest-pinned refs back to the repo after each publish. Rejected: the strongest pin, but it needs a bot pushing to protected main (or an auto-merged PR) and a trigger-loop guard, and every publish then produces a second deploy-triggering commit. The versions.env default plus the deploy-side wait gets "the pin only advances behind a published, pullable image" without any of that machinery; digests remain the documented strict contract for hand-set refs.

  • A startup-once preflight with the race documented ("restart after deploys"). Rejected: even with the deploy-side wait, registry blips and dispatch re-releases leave windows where only time heals the pull; an operator instruction that must be followed after every such window is an outage schedule. The retry loop is small and uses the gate exactly as designed.

  • image exists-first preflight kept as-is. Rejected: with moving version tags as the default refs, exists-first freezes every host on its first-ever pull of a tag — a labctl refresh would reach new hosts only. Pull-first with a cached fallback keeps the freshness of pull-always and the availability of exists-first.

Consequences

  • A plain services.lab.enable = true deployment is container-capable after its first successful preflight — podman, delegation, subids, runtime dir, and published-and-pullable tools refs all present with zero container config. Flipping a repo to repos.runner = container is now a UI action, not a host-provisioning project.
  • Host-only deployments become the explicit case: container.enable = false, zero-diff guaranteed by the checks.
  • CI stays cheap on ordinary merges. Images rebuild only on containers/** changes (or dispatch); the deploy gains one usually-instant registry check.
  • A versions.env bump is the one merge shape that serializes deploy behind a ~15-minute image build. The deploy's wait step absorbs it; a publish failure blocks the pin bump for that rev — treat a red agent-tools publish as a broken build.
  • The agent-tools package must stay publicly readable. Publish verify, deploy wait, and host preflight all enforce it loudly at their seams.
  • Existing hosts pick up podman and subid ranges on their next deploy. Operators with pre-existing subid allocations (or force-disabled podman) must reconcile once — override subIdRange, or set container.enable = false.