Skip to content

Containers become transient scopes under the lab user's lingering user@ manager: cgroup v2 gates PID migration on the common ancestor, so an unprivileged lab can never place a container in another unit's subtree

On 2026-07-26 the ADR-0059 holder layout — merged to main (#238) and deployed on dev-new — failed on first contact: every container spawn died at podman create, conmon/crun stderr write to /sys/fs/cgroup/system.slice/lab-payload.service/payload/libpod-<id>/cgroup.procs: Permission denied. The deployed tree was exactly what ADR-0059 specifies; nothing was misconfigured. The failure is cgroup v2's delegation containment rule: to migrate a PID between cgroups, a non-root writer needs write access to cgroup.procs of the common ancestor of source and destination — not just the destination. crun runs inside lab.service; the target is lab-payload.service; the common ancestor is system.slice, whose cgroup.procs is root-owned — EACCES on every attach. Three live probes pinned it: (1) a lab-user write of a session PID into a fresh lab-owned cgroup under payload/ → Permission denied; (2) the identical write as root → succeeds; (3) a lab-user move of a PID whose source is already inside the lab-payload.service subtree → succeeds. Probe 3 is the smoking gun: destination permissions are fine; only the cross-delegation ancestor check fails.

The general form, so nobody re-attempts a holder variant: whoever spawns containers must share the delegated subtree, and whoever shares the subtree cannot safely restart with survivors — only root/systemd escapes both horns. ADR-0058 (payload inside lab.service → restart-EBUSY) and ADR-0059 (payload in a separate unit → spawn-EACCES) are two sides of the same kernel wall. The fix is the fallback both ADRs documented and rejected, promoted to primary: stop performing cgroup placement as an unprivileged process at all — let systemd do it.

/sys/fs/cgroup/
├── system.slice/
│   └── lab.service/                 plain unit cgroup — lab + tmux + every host
│                                    process; no Delegate=, no subgroups, no
│                                    cgroup machinery of any kind
└── user.slice/user-<uid>.slice/
    └── user@<uid>.service/          the lab user's lingering manager
        └── user.slice/
            └── libpod-<id>.scope/   one transient scope per container;
                                     --memory/--pids-limit bind here as
                                     MemoryMax/TasksMax

The pins, decided (2026-07-26, issue #239):

  • users.users.lab.linger = true (NixOS module, container.enable-gated) gives the lab user a permanent user@<uid>.service manager, up from boot and independent of lab.service. lab.service itself stays a system service running as lab — the linger adds a manager beside it, it does not move lab into a user session.

  • Podman stays rootless but switches to --cgroup-manager=systemd in every shape — run pane, login pane, non-interactive CLI poke; one renderer, podmanx.RunArgv — rendered unconditionally, replacing --cgroup-parent=…/payload --cgroup-manager=cgroupfs. Each container becomes its own transient libpod-<id>.scope created via the lab user manager; the user manager forwards the cross-unit attach to PID 1 (AttachProcessesToUnit), so the ancestor rule never applies — the one writer that places PIDs is systemd itself. --memory/--pids-limit land as per-scope caps (MemoryMax/TasksMax) on the scope's own cgroup.

  • lab wires the manager itself: XDG_RUNTIME_DIR=/run/user/<uid> and DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/<uid>/bus, set process-wide at startup. The uid is unknown at nix eval time, so the unit cannot carry them; the tmux env baseline forwards both into every pane, and at adoption of a surviving pre-fix tmux server its server-global env is re-pinned to the current baseline so stale /run/lab values cannot leak into new panes. This reverses ADR-0057's XDG_RUNTIME_DIR=/run/lab decision: podman's runroot/tmpdir move to /run/user/<uid>, whose lifetime linger owns (up from boot, independent of lab.service), so the RuntimeDirectory=lab + RuntimeDirectoryPreserve machinery is retired. Migration cost, accepted: podman's local db records the old runroot/tmpdir, so hosts that ran the /run/lab layout hit database configuration mismatch on first use — the one-time fix is podman system reset --force as the service user (wipes local containers/images; the preflight re-pulls tools images each boot and dev images re-pull at spawn, so nothing of value is lost).

  • Deleted wholesale: the lab-payload.service holder, the lab-cgroup-hygiene oneshot, podmanx.setupCgroups, the per-spawn restart-safety guard (Cgroups.Verify), and Delegate=/DelegateSubgroup= on anything lab manages. Each failure mode disappears structurally instead of being pinned: no unit lab manages carries delegation, so a lab.service restart can never EBUSY; containers survive lab restarts by construction (independent scope units, not fellow travelers in lab's cgroup); and user@lab is systemd's own machinery — never restarted by nixos-rebuild, and stopping it kills its whole subtree, so it has no survivor-EBUSY mode either.

  • Preflight becomes a real spawn probe — the old one was false-green. It verified controllers and ownership but never performed a PID migration, the one operation that fails. New checks: (a) user-manager — writability of /run/user/<uid>; retryable, because logind brings user@ up asynchronously at boot and lab cannot order on it (the uid is unknown at eval). (b) spawn-probe — podman create + podman init a probe container named lab-preflight-probe (deliberately not labrun-*: the orphan sweep would race it) off the first resolvable agent-tools image with --memory 64m --pids-limit 16, command /bin/labctl (the static binary every agent-tools image carries; init pauses before exec, so nothing in the image ever executes and the images' never-run contract holds in spirit); assert the container landed in a libpod-*.scope under the user manager and that memory.max/pids.max in that scope cgroup read exactly 67108864/16; then rm. This preserves #205's no-false-green property for real, regardless of layout.

Status

Accepted. Diagnosed and decided 2026-07-26 from the dev-new spawn-EACCES incident (issue #239). Supersedes ADR-0059 entirely — merged to main (#238) and deployed on dev-new 2026-07-26, it is superseded in place having never worked in production: every spawn after its deploy failed — and the remaining argv/manager pins of ADR-0058: the --cgroup-parent + --cgroup-manager=cgroupfs rendering and the empty-parent fail-loud property are retired with the layouts they served. This is the promotion of the linger + user@ fallback both ADRs recorded; it also reverses ADR-0057's runtime-dir decision (/run/lab → /run/user/<uid>), while everything else those ADRs decided — the containerized login/CLI surfaces (0057), the one-renderer podmanx.RunArgv shapes and KillMode=process session survival (0058) — stands. Migration: survivor containers under the old cgroup locations keep running uncapped until natural exit (the same accepted stance as the 0058/0059 migrations); on dev-new, prune the ~1460 empty libpod-* leftover dirs under the old holder's payload/ (the holder unit disappears from the config on deploy; its cgroup dies with its last survivor), and the emergency unblock used there — chown lab /sys/fs/cgroup/system.slice/cgroup.procs, satisfying the ancestor check until systemd re-realizes the slice — is retired by this fix. Acceptance: (1) provider login round-trips; (2) agent run containers start with --memory/--pids-limit verifiably applied to their scope cgroup; (3) systemctl restart lab with live containers — lab returns clean, containers survive, new spawns work; (4) preflight fails loudly and actionably when the user manager is unreachable; (5) lab-payload.service, lab-cgroup-hygiene, setupCgroups, and the spawn guard are gone from module and code.

Considered options

  • Patch the holder: keep lab-payload.service and fix its restart. Impossible — the #237 restart-EBUSY trap still exists inside the holder: DelegateSubgroup= never applies to the main-process spawn (the systemd v260 exec_params_needs_control_subcgroup receipt in ADR-0059), so any holder restart with surviving containers EBUSYs; "never restart it" is a pin, not a fix.

  • podman system service (REST API) doing the placement. No help: the invariant is that an unprivileged process can only attach PIDs into a delegated subtree it already lives in, so the API service would have to live inside the payload unit — just another spawner carrying the same restart trap, and worse: it idle-exits, is socket-activated, and its binary changes every deploy.

  • A custom spawner daemon inside the holder. Same analysis, same trap — plus an un-updatable binary pinned inside a never-restarted unit.

  • Keep --cgroup-manager=cgroupfs under the user manager. Rejected: still an unprivileged cross-cgroup write — it misses the point that placement, not location, is the problem.

  • A root-owned attach helper. Rejected: a new privilege surface to build and maintain against a problem systemd already solves (AttachProcessesToUnit).