Skip to content

One priority-ordered spawn pass replaces the two loops that raced for the AFK/autoland fleet's instance cap — drain the pipeline before filling it, one cap consumer by construction, and lander outcomes leave the three-strikes counter untouched

Before this change the fleet launched runs from two independent goroutines. The scheduler tick (its selection sweep) launched new AFK work; the reaper tick's autoland sweep (AutolandOnce, added by #181) launched landers. Each read liveCount < EffectiveCap(repo) against its own liveness snapshot and each launched, so whichever tick fired first took the slot. Two things followed. The cap could be over-subscribed — two loops racing to consume one bounded resource is a race by construction, not an edge case that finer timing avoids. And the lander had arbitrary priority, not high priority: whether a claim PR got validated before a fresh issue was started came down to tick alignment, nothing more. #182 makes this strictly worse by adding fix runs as a third spawner — a third reader of the same cap, a third launcher racing the other two.

The move is to collapse the launch decision into one serialized pass. Producers gather launch candidates and never launch; a single pass merges the lists, sorts by pipeline stage, and spends the cap down the list. The two loops keep their other duties — reaping, the runtime sweep, the scheduler's fill heartbeat — and only the spawn decision merges. Doing this before #182 turns the fix-forward loop into one new candidate producer instead of a modification of a racing structure. Autoland ships default-off (#181/#184), so the race is latent: this clears a correctness debt before the third spawner lands, it is not a live production defect.

The pins, decided:

  • One priority-ordered spawn pass owns every launch; drain the pipeline before filling it. SpawnOnce is the fleet's single spawn decision point: one repo listing, one liveness snapshot, one shared forced-auth memo feed the producers (landerCandidates, newWorkCandidates), and their gathered candidates go through spawnPass, which stable-sorts by SpawnStage and launches down the list while the repo is under its effective cap. The order is StageLander < StageFix < StageNewWork — a lower rank sits further down the pipeline and launches first. The rationale is queueing, not aesthetics: finishing an in-flight claim (validate its PR, or repair a rejected one) drains the pipeline; starting a fresh issue fills it. Without the ordering a repo at cap can spend every slot opening new PRs while validated-but-unlanded ones queue up behind them — the pipeline grows at the head and never empties at the tail. So work already in flight outranks starting new work, at every stage.

  • One consumer of the cap, by construction — the race is removed, not narrowed. The pass is serialized under spawnMu; producers gather and never launch; spawnPass is the sole reader of EffectiveCap in the launch decision. The old race was structural: two goroutines each comparing liveCount < EffectiveCap against its own snapshot and each launching, so a smaller window between read and launch would only shrink the odds, never close them. Serializing the one pass means concurrent scheduler/reaper ticks (and the HTTP kicks) queue on the lock instead of interleaving their snapshots — two cap consumers become one. The authoritative per-launch guards inside launch and LaunchLander stay: they re-prove fresh liveness under the engine lock and remain the backstop on that same path. The pass's snapshot check (liveCount >= EffectiveCap) is a cheap load-shedding veto, never the enforcement — the lock-held launch is.

  • Priority is data — a SpawnStage rank on the candidate — not control flow spread across files. A candidate carries its stage, and stage is the only sort key. StageFix is already reserved between StageLander and StageNewWork with no producer emitting it. So #182's fix-forward loop becomes exactly one new producer appended alongside the two gathers, emitting candidates at StageFix; the sort owns the order and the pass needs no edit. That is the whole point of doing this first: a third spawner would re-enter the race this ADR removes, a third producer cannot — it hands its candidates to the one consumer like the other two.

  • Candidates are re-derived every pass from polled state — deliberately not a queue. Each pass re-lists repos, snapshots liveness once, and re-walks polled forge state (the pull listing, its reviews, its verdict-marker comments) plus the runs store to gather candidates; nothing persists between passes. ADR-0015's poll-only model stands: a crash mid-pass loses nothing, because the next pass re-derives the identical candidate set from the same durable facts — the PR's verdict state and the runs store — that survive a restart. A candidate is a momentary read, not an entry in a durable work list.

  • Lander outcomes never move the AFK three-strikes counter — in either direction. reapRun's failure switch was kind-agnostic (its own doc enumerated only the AFK kinds, so the coupling read as unintended), which is wrong for a lander two ways. A lander death or timeout would IncrementRepoFailures; at PauseThreshold the repo's AFK auto loop pauses and manual starts 409 — lander flakiness silently suspending unrelated AFK work. And a lander posting reject reaps as OutcomeSuccess → ResetRepoFailures, so a lander correctly rejecting a broken AFK PR clears the very strikes those broken runs accumulated — the brake re-armed by exactly the evidence that should trip it. Under fix-forward that compounds: the pipeline would feed itself, each rejection wiping the record of the failure that caused it. So a lander outcome (and a fix run's) touches consecutive_failures in neither direction; only manual and auto AFK run outcomes feed it, and their accounting is unchanged.

  • Landers get no counter at all, and the budget clock stays shared. Severing the wrong coupling is not the same as giving landers a right one. Nothing consumes a lander failure count today — no lander-specific pause, no escalation keyed off it — so none is added; the pinned requirement is only that a lander outcome never moves the AFK counter. The budget clock stays shared deliberately: a lander is still a run bounded by its budget_deadline like any other, and that coupling was never wrong. Only the failure counter separates.

Status

Accepted. Settled via issue #185. Restructures the spawn engine that ADR-0048's autoland contract is read by — the contract itself (verdict-marker PR comments, the state-derived poll-only read, the shared validation core, default-off) is unchanged; only where the launch decision lives has moved, from two racing sweeps into one pass. Keeps ADR-0015's poll-only model intact: candidates are re-derived every pass from polled state, no queue and no dispatch between engine and run. Leaves the three-strikes pause semantics (ADR-0007, ported by ADR-0010: never an automatic un-pause) unchanged for the AFK kinds — only the lander's accidental coupling to that counter is cut. Precedes #182: the fix-forward loop lands as a producer emitting candidates at the reserved StageFix rank, an addition to the gather rather than a modification of the pass.

Considered options

  • Keep the two loops, add finer locking around the cap read. Rejected: a shared lock around "read liveCount, compare to EffectiveCap, launch" narrows the window between read and launch but does not remove the race — two loops remain two consumers of one bounded resource, and every future spawner widens the surface again. One serialized pass with a single cap reader removes the race by construction; that is a categorically stronger invariant than a smaller window, and #182 would have re-opened the window regardless.

  • A persistent candidate queue the loops push into and a spender drains. Rejected: it violates ADR-0015's poll-only model. A queued candidate is state that does not survive a restart, and it forks the source of truth away from the PR's verdict state plus the runs store, which do survive. Re-deriving candidates every pass is re-entrant by construction; a queue is one more thing to lose, and it buys nothing the re-derivation cannot already compute.

  • Leave the UnderCap term in the two spawn predicates. Rejected: that keeps two cap consumers — the predicate and the pass — which is the original race under a new name. ShouldLaunchAuto and ShouldSpawnLander drop their cap term entirely; each now weighs only its own load-bearing facts (opted-in, not paused, work waiting, nothing already in flight), and cap enforcement is the pass's alone. A predicate that still vetoed on the cap could disagree with the pass about who gets the last slot.

  • Give landers their own failure counter. Rejected for now: nothing reads it. A separate lander-strikes brake is worth adding only when something acts on it — a lander-specific pause, an escalation keyed off repeated lander deaths — and adding the column and the writes before a consumer exists is speculative state that can drift untested. The pinned requirement — a lander outcome never moves the AFK counter — holds with no lander counter at all; add one when a consumer arrives.

Consequences

  • Both producers now run on both cadences. Merging the spawn decision means every SpawnOnce pass gathers from both producers regardless of which loop invoked it: landers are considered on the scheduler tick, and new work is considered on the reaper tick (right after the reap) and on the HTTP toggle-on/reset kicks. Before, each loop ran only its own sweep. This is the intended shape — the reaper tick frees run rows and cap slots, so filling on that same tick makes drain-then-fill literal and a fresh PR gets its lander within one tick — but it does mean lander forge reads now also happen on the scheduler cadence. The at-cap production veto in each producer (the liveCount >= EffectiveCap pre-filter, sitting exactly where each old sweep's UnderCap veto sat) keeps an at-cap or autoland-less fleet from paying for reads it cannot use.

  • Autoland's whole-sweep at-cap stop became a per-repo floor-raise. The old autoland sweep stopped the entire sweep on the first ErrOverCap — correct only when every repo shares one global cap. Under per-repo cap overrides (D12c) a global count at cap no longer implies every repo is at cap, so spawnPass skips only the candidate whose repo just proved at-cap and raises the stale snapshot to that repo's cap: a sound lower bound (the locked launch proved the fresh count is at least that) that lets later per-repo vetoes fire without another lock-held launch, and still reproduces a whole-pass stop whenever every repo shares the one global cap. The behaviour is strictly more correct under per-repo overrides, not merely preserved.