Skip to content

A static lab-payload.service holder unit owns the delegated payload cgroup: DelegateSubgroup= never protects the main-process spawn, so lab.service can never delegate

On 2026-07-26 the ADR-0058 layout — grilled and validated green the day before — recurred on dev-new (systemd 260.2): every lab.service restart after the ADR-0058 deploy failed with Failed to spawn executor: Device or resource busy (result resources) and restart-looped past counter 700, while the agent containers, tmux, conmon and pasta survived exactly as designed. The on-disk cgroup tree was bit-for-bit what ADR-0058 specifies, and the lab-cgroup-hygiene oneshot ran green before every one of those 700 attempts. The layout was correct and the outage was still total — because ADR-0058's core premise, "systemd spawns lab into main/", is false.

DelegateSubgroup= does not apply to the unit's main process. In systemd v260 (unchanged on main), src/core/execute.c:

int exec_params_needs_control_subcgroup(const ExecParameters *params) {
        /* Keep this in sync with exec_params_get_cgroup_path(). */
        return FLAGS_SET(params->flags, EXEC_CGROUP_DELEGATE|EXEC_CONTROL_CGROUP|EXEC_IS_CONTROL);
}

All three flags must be set, and the ExecStart= main process is not a control process (EXEC_IS_CONTROL is clear), so exec_spawn() hands the unit cgroup root — not main/ — to clone3(CLONE_INTO_CGROUP); sd-executor migrates itself down into main/ only after it has already been spawned into that root. And posix_spawn_wrapper (src/basic/process-util.c) retries the spawn without the cgroup flag only on ENOTSUP/EPERM, never on EBUSY. So the moment lab enabled +memory +pids on its unit root's subtree_control and anything survived a stop — KillMode=process keeping the sessions alive across deploys is the product feature — cgroup v2's no-internal-process rule rejected the next CLONE_INTO_CGROUP into that root with EBUSY, unretried, on every subsequent start. A fresh boot worked only because the root's subtree_control is empty at spawn time and the controllers get enabled by lab's own preflight moments later — which is precisely why ADR-0058 validated green on first start and then wedged on the next deploy. Any design that keeps controllers enabled on lab.service's own cgroup root across a restart is structurally dead: the root cause is in systemd's main-process spawn path and EBUSY is not retried there, so the fix cannot live inside lab.service at all. Recovery on the night was manual — clear payload/, then the unit root's subtree_control bottom-up. Source receipts: v260 execute.c, main execute.c, v260 process-util.c.

The fix moves the delegation out of lab.service entirely, into a second unit that is never restarted and so can safely hold a delegating cgroup root on lab's behalf:

/sys/fs/cgroup/system.slice/
├── lab.service/                 plain, UNDELEGATED unit cgroup — lab, tmux, all
│                                host processes; its subtree_control MUST stay
│                                empty forever (else the next restart EBUSYs)
└── lab-payload.service/         "holder" unit — delegated root (Delegate=yes,
    │                            User=lab), never restarted; subtree_control
    │                            carries +memory +pids (enabled by lab preflight)
    ├── main/                    holder's own sleep process (DelegateSubgroup=main
    │                            keeps the delegated root proc-free)
    └── payload/                 every container via --cgroup-parent
        └── libpod-*/            per-container leaves; caps bind here

The pins, decided (grilled 2026-07-26 with issue #237; the recurrence forced the diagnosis ADR-0058's first-boot green had hidden):

  • A static lab-payload.service holder unit owns the delegated payload cgroup. NixOS module, container.enable-gated, an implementation detail — no new options: Type=exec + ExecStart=sleep infinity, User/Group = the lab service user, Delegate=yes, DelegateSubgroup=main, KillMode=process, and NixOS restartIfChanged=false. It is never restarted — that is the whole point: the executor-spawn trap needs a restart to fire (a fresh spawn into a controller-armed root), so a unit that never restarts can safely keep a delegating root indefinitely. DelegateSubgroup=main is still correct here — not as spawn protection (it never was that) but so the holder's own sleep sits in main/, keeping the delegated root process-free so controllers can be enabled on it. Type=exec means the start job completes only after sd-executor has migrated into main/, so lab's After= ordering never races the holder's cgroup setup. KillMode=process so stopping the holder never takes the containers down. Delegate=yes + User= chowns the delegated subtree to the lab user — what lets rootless lab/podman write there without new privileges.

  • A manual holder restart while containers survive hits the same EBUSY — documented, not fixed. The holder's root still delegates and its cgroup is still populated by the surviving containers, so restarting it re-runs the exact wedged spawn: the unit fails to start, the containers keep running untouched. Recovery is stop the holder, wait the containers out (or stop them), start the holder, then restart lab so preflight re-establishes the layout. Deploys never trigger this (restartIfChanged=false pins the holder across every rebuild); it is only reachable by a hand-typed systemctl restart lab-payload.service, and the failure is loud and self-describing rather than silent corruption.

  • lab.service loses Delegate=/DelegateSubgroup= entirely and gains Wants=/After= on the holder. lab, tmux and every host process now live directly in the plain, undelegated lab.service cgroup, whose subtree_control must never carry an entry — that empty-subtree invariant is the thing that keeps the next restart's spawn legal. Wants=, not Requires=: a missing or failed holder degrades to an actionable preflight refusal naming lab-payload.service, never a systemd-vetoed lab start. KillMode=process and RuntimeDirectory/RuntimeDirectoryPreserve are unchanged from ADR-0058 — session survival across a lab restart is untouched.

  • Preflight (podmanx.setupCgroups) hardcodes the holder path — no more /proc/self/cgroup derivation. The payload cgroup is /sys/fs/cgroup/system.slice/lab-payload.service/payload, shipped in lockstep with the module rather than read from lab's own cgroup (lab is no longer in the delegated tree, so there is nothing to derive). Preflight then: confirms the holder root is writable by the service user — the Delegate=yes + User= chown signal — else the new cgroup-holder check fails, naming lab-payload.service as the unit to bring up; adopts any stray pid still sitting at the holder root into the holder's main/ (covering a systemd < 254 that silently ignored DelegateSubgroup=); enables +memory +pids on the holder root's subtree_control and creates payload/, then confirms the controllers actually arrived there (the cgroup-delegation check, name kept from ADR-0058). The old cgroup-layout check — lab-in-a-delegated-subgroup — is retired; lab being in a delegated subgroup is exactly the failure mode this ADR removes.

  • The per-spawn restart-safety guard (cgroup-restart-safety, name kept) retargets to the two invariants that now matter. It runs at preflight and again before every container spawn, hard-failing, and re-checks: (a) lab.service's own cgroup subtree_control is empty — nothing (a podman heuristic, a stray controller-enable) may ever re-arm the trap on the undelegated unit; (b) payload/'s cgroup.controllers still carries memory + pids — a restarted or stopped holder would otherwise mean silently uncapped containers, the ADR-0058 / #205 no-false-green lesson. Refusal text says to restart lab to re-establish the layout. Refusal shapes per surface are unchanged (*BadRequestError for runs, provider.ErrLoginUnavailable for login panes, plain could-not-run for CLI pokes).

  • The root lab-cgroup-hygiene oneshot is RETARGETED, not retired. Ordered before every lab start, it now clears every cgroup.subtree_control under /sys/fs/cgroup/system.slice/lab.service including the unit root's own, bottom-up (a parent cannot drop a controller while a child still delegates it — the ordering the manual recovery tripped over), and prunes emptied child dirs. This is simultaneously the migration off the ADR-0058 layout — on the first post-upgrade start the old root still carries +memory +pids while surviving main/ processes keep the cgroup alive, i.e. the armed trap itself — and standing insurance against anything re-arming the undelegated unit between restarts. Accepted migration cost: pre-0059 survivor containers under the old lab.service/payload/ lose their caps when this clears the old root — they are already outside the new verified subtree, the same posture ADR-0058 accepted for its migration (uncapped-until-natural-exit beats killing live sessions to relabel them).

  • Everything else from ADR-0058 stands. --cgroup-parent=<…>/payload + --cgroup-manager=cgroupfs in every shape — run pane, login pane, non-interactive CLI poke, one renderer (podmanx.RunArgv); the empty-CgroupParent-renders-no-flags fail-loud property; and the KillMode=process / RuntimeDirectoryPreserve restart-survival contract. Only where the delegating root lives changed — from lab.service itself to the lab-payload.service holder — and with it the preflight/guard/hygiene targets above.

Status

Superseded by ADR-0060 — in place, having never worked in production. The 2026-07-26 spawn-EACCES incident proved the cross-unit attach this ADR requires impossible for an unprivileged lab: cgroup v2 gates a PID migration on write access to cgroup.procs of the common ancestor of source and destination — here system.slice, root-owned — so every podman create after the deploy died with Permission denied, and the holder never spawned a single container in production. ADR-0060 moves cgroup placement to the lab user's lingering user@ manager (--cgroup-manager=systemd, one transient scope per container) and deletes the holder, the hygiene oneshot, setupCgroups, and the spawn guard. The text below is retained as the historical record.

Accepted. Diagnosed from the 2026-07-26 dev-new recurrence (issue #237) and grilled the same day; the diagnosis is backed by the systemd v260 source receipt above (exec_params_needs_control_subcgroup requires EXEC_IS_CONTROL, which the main process never carries, so DelegateSubgroup= cannot move the main-process spawn off the unit root — unchanged on main). Supersedes ADR-0058's core placement: the in-unit main/ + payload/ split and lab.service's own Delegate=/DelegateSubgroup= are retired — the delegation moved to the never-restarted lab-payload.service holder unit, lab.service is now plain and undelegated, and its subtree_control must stay empty forever. ADR-0058's other decisions carry forward unchanged and are not re-litigated here: the --cgroup-parent + --cgroup-manager=cgroupfs argv (one renderer, podmanx.RunArgv), the empty-parent fail-loud property, controller-verification-not-listing, and KillMode=process / RuntimeDirectoryPreserve session survival. The module invariants live next to the holder unit in nix/module.nix; the layout mechanics and the retargeted checks in internal/podmanx/cgroup.go. Fallback, unchanged from ADR-0058 and still rejected here: rootless podman under a lingering user@lab manager (containers in transient scopes in a different unit) — kept as the documented escape hatch so nobody re-grills the option space from scratch (see ADR-0058's fallback paragraph).

Considered options

  • Keep the delegation on lab.service and protect the main-process spawn some other way. Structurally impossible: the root cause is in systemd's own spawn path — exec_spawn() targets the unit cgroup root for the main process regardless of DelegateSubgroup=, and posix_spawn_wrapper does not retry EBUSY — so there is no unit-side directive that both delegates controllers on the root and keeps the next restart's spawn legal. This is what ADR-0058 tried; it is the failure this ADR exists to undo.

  • Patch or version-pin systemd, or file it upstream. Rejected as the fix — out of scope and a moving dependency. The holder unit works on stock systemd (the behavior is consistent v254→main), so it needs no patched systemd. An upstream issue or documentation clarification is a nice-to-have, not a blocker, and is not what ships here.

  • Podman-canonical: users.users.lab.linger = true and a real user@ manager. Still rejected, unchanged from ADR-0058: a permanent user manager for a system user, containers leaving lab's delegated subtree (killing the verify-the-exact-subtree property and Delegate='s purpose), and it re-opens ADR-0057's XDG_RUNTIME_DIR=/run/lab + RuntimeDirectoryPreserve machinery. Kept as the documented fallback above — the holder unit is a smaller move that keeps the container caps inside a subtree lab owns and preflight verifies.

  • Retire the hygiene oneshot entirely, now that lab.service never delegates. Rejected: migrating off the ADR-0058 layout requires clearing the old lab.service unit-root subtree_control while surviving main/ processes keep that cgroup alive — the armed trap — and an undelegated lab.service cannot clear its own root from an ExecStartPre (that process spawns into the same wedged cgroup). Only a unit outside lab.service's cgroup can do the root-side clear before the attach, which is exactly what the retargeted oneshot is.