Skip to content

Server-owned credential authority: one refresher per provider grant, a blind periodic poke against the master store, immediate fan-out, and eager adopt-back checked before every wipe

Issue #202 gave every run a private instance HOME, and InjectCredentials snapshot-copies whatever the machine's master credential store holds into each one at spawn — a one-way copy, with no write-back. That was fine as long as nothing ever rotated the copy in place. Both Anthropic's and OpenAI's CLIs do exactly that: the OAuth access token is short-lived (~8h for claude), and the CLI refreshes it unattended against its on-disk credential file, receiving a new access+refresh token pair — a new token family — and invalidating the refresh token every other holder of the old family still has. With one refresh token shared verbatim across the master store and every instance's snapshot, whichever CLI refreshes first — master or any live instance — logs out everyone else: the host reads logged-out, the spawn gate (AuthStatus(ctx, force=true)) refuses new runs, and wipe-on-stop can delete an instance HOME holding the only token family still valid. This is not hypothetical: live on the dev host 2026-07-24, three independently-refreshed families were alive at once, each rotation orphaning every other holder.

The grilled triage session (issue #222, 2026-07-24) settled one invariant and the design that enforces it: exactly one refresher runs per provider grant; every instance is a consumer of the resulting token family, never a refresher of its own. Lab core owns that invariant without ever speaking a provider's OAuth protocol — it pokes a CLI, watches a file, and copies bytes.

The pins, decided:

  • Refresh clock — a blind periodic poke. A lab-owned loop periodically runs the provider's own refresh-trigger CLI command on the host, against the master store, and lets the CLI itself decide whether the token is near expiry and rotates it. Lab detects a resulting rotation purely by the credential file changing — an opaque signature comparison, never by parsing token contents; a poke that rotated nothing and one that rotated the whole family are indistinguishable to lab by design. For claude-code the probe-confirmed trigger is a real (cheap) API call, claude -p --model haiku under the master CLAUDE_CONFIG_DIR — probed live 2026-07-24 by forging the master's expiresAt 60s into the past and observing the store rotate in place. The codex analogue (codex exec, non-interactively) is a candidate pending its own live verification.

  • Fan-out on rotation — immediate. Any change to the master's signature re-injects into every live instance HOME through the existing InjectCredentials seam — its single call site (the launch path) is preserved, and instance HOMEs are host paths lab already writes to even for container runs (ADR-0052). Probed live: rotation invalidates the previous family's access tokens within minutes, so fan-out has to directly follow detection, not trail it. The payoff compounds: copies kept fresh are never near their own expiry, so instances never have a reason to self-refresh, and the design never has to survive a thundering herd of simultaneous refreshers.

  • Adopt-back — eager, on detection. The safety net for the case fan-out doesn't fully close: if an instance's CLI self-refreshes anyway, the scan detects that instance's signature diverging from the signature core last injected into it, adopts that family into the master immediately, and fans it back out to every other instance. When several homes diverge in the same scan, the tie-break is newest credential-file mtime wins.

  • Wipe races — an adopt-check before every wipe. The signature core last injected into a run is persisted on the run row (runs.cred_sig), so it survives a restart. Every path that can destroy an instance HOME — Stop, Start's rollback, and the boot/runtime sweep that GCs orphaned homes after downtime or mid-flight — synchronously compares the run's current signature against cred_sig and adopts a self-refreshed family into the master before the wipe runs, so a wipe can never be the thing that destroys the fleet's only remaining valid family.

  • Seam shape — three new methods on provider.AgentProvider, alongside InjectCredentials. RefreshCredentials(ctx context.Context) error drives the provider's own CLI to poke the master store; a caller gates the call on the store being populated and bounds it with a context timeout, since it is a live subprocess. CredentialsSig(home string) (sig string, mtime time.Time) is an opaque, adapter-private change-detector over the credential file(s) under home — "" means the machine's master store, following the SpoolSig precedent (ADR-0020: existence + mtime + size digest); core compares sigs for equality only and never parses them, and the returned mtime is exposed as typed filesystem metadata specifically so core can own the newest-wins adopt tie-break without ever learning the adapter's file layout. AdoptCredentials(instanceHome string) error is InjectCredentials' reverse: an atomic (tmp-file-plus-rename, within the master directory) copy of an instance's credential file(s) back into the master store, so a live host-side CLI reading the master store never observes a torn file. Core owns the schedule, the signature comparison, the fan-out ordering, and the adopt tie-breaks; each adapter owns every file path and its provider's own CLI recipe — lab core never parses token contents, on either side of the seam.

Rejected alternatives:

  • A credentials file shared (symlinked or hardlinked) across every home. claude derives its credentials path from CLAUDE_CONFIG_DIR, which must stay per-run for isolation (issue #202); and the CLI's own refresh is a tmp-file-plus-rename over that path, which replaces whatever inode a symlink or hardlink pointed at — the same agent.sock inode lesson ADR-0052 already paid for (a bind mount, or a link, pins the inode that existed at setup time; the CLI's rename creates a new one). Sharing the file also reopens the exact instance-HOME isolation seam issue #202 closed.
  • Sync-back alone, with no refresh loop. Every holder still shares one expiry clock, so the common case becomes a thundering herd of concurrent self-refreshes at T+8h, all racing to rotate the same shared refresh token — sync-back only tells core who won, it does nothing to stop the race.
  • Adopt-back triggered only once the master fails. This leaves a window — up to the remaining lifetime of the master's own (already-invalidated) token — during which the fleet is holding a family the CLI knows is dead but core does not yet know to replace. Eager adopt-back on divergence closes that window at detection time instead of at master-failure time.
  • Probing the server-side refresh-token grace window. A grilled-but-dropped idea: ask the provider's token endpoint how much slack it gives a refresh token after a newer sibling has rotated it, and use that as a deadline. Rejected because probing it means spending a live refresh token specifically to measure its own grace window — the measurement is destructive. Eager adopt-back removed the design's dependency on ever knowing the answer.
  • A server-side credential proxy, anticipated as a possible later evolution when issue #202 first shipped InjectCredentials' single call site. Still open for later; this design is the snapshot-plus-fan-out model that call site was built to keep replaceable, not a replacement of it.

Status

Accepted. Settled via issue #222 (2026-07-24, the grilled credential-authority design), following live evidence on the dev host of three independently-refreshed token families alive at once. Builds on issue #202 (the per-run instance HOME and InjectCredentials's snapshot copy — this design adds the write-back path that call site was deliberately kept singular for) and ADR-0052 (the instance-HOME mount and the agent.sock inode lesson the shared-file alternative re-triggers). Follows the SpoolSig precedent from ADR-0020 for CredentialsSig's existence+mtime+size shape. Implemented with issue #222: the provider seam (RefreshCredentials, CredentialsSig, AdoptCredentials on provider.AgentProvider, doc-commented at the "credential-authority seam" in internal/provider/provider.go) for both claude-code and codex, the runs.cred_sig column, the core rotation loop (internal/credrotate: schedule, fan-out ordering, adopt tie-breaks), and the wipe-site adopt-checks (the pre-wipe hook on instancehome.Manager, covering every wipe and sweep path). The codex adapter's refresh trigger remains a candidate recipe pending its own live probe (internal/compat/codex/compat.md §11).

Considered options

  • A shared credentials file across every instance HOME. Rejected: incompatible with per-run CLAUDE_CONFIG_DIR isolation (issue #202), and the CLI's tmp-file-plus-rename refresh defeats any link-based sharing the way a container restart defeats a socket-file bind (ADR-0052).
  • Sync-back with no active refresh loop. Rejected: leaves every holder on one shared expiry clock, so the common case is several CLIs racing to refresh the same token simultaneously — sync-back reports the outcome of that race, it does not prevent it.
  • Adopt-back gated on a detected master failure. Rejected: the fleet still runs on a doomed family for however long the master's already-invalidated token takes to be noticed dead, instead of the family being adopted the moment a live instance proves it still works.
  • Probing the provider's refresh-token grace window server-side. Rejected: the only way to measure it is to spend a live refresh token, which is destructive to the very credential the design exists to protect. Eager adopt-back made the answer unnecessary.
  • A server-side credential proxy in place of the copy-based seam. Deferred, not rejected: issue #202 built InjectCredentials's one call site so this could replace the copy wholesale later without touching a caller. This design is the snapshot-plus-fan-out model that seam anticipated, not the proxy itself.

Consequences

  • The provider contract grows three methods, and every adapter must implement all three or fail conformance. RefreshCredentials, CredentialsSig, AdoptCredentials join InjectCredentials on provider.AgentProvider; providertest.Conformance asserts their error-guard and empty-master/empty-sig contracts (issue #222) for every registered adapter, claude-code included.
  • runs.cred_sig is a new persisted column, nullable for pre-#222 rows. It is the adopt-check's baseline across a restart — a run row with no stamped signature (a pre-upgrade row, or a launch with no master credentials at spawn) skips the adopt-check rather than false-positiving a divergence.
  • The rotation loop is a new, always-on background consumer of the master store and every live instance HOME. It runs regardless of whether any instance is actively using its credentials, at whatever cadence keeps pokes well inside the ~8h access-token lifetime; a poke against a logged-out host (CredentialsSig("") == "") is skipped, never attempted.
  • Wipe call sites (internal/instance/stop.go, launch.go's rollback, and the boot/runtime sweep that GCs orphaned homes) each gain a synchronous adopt-check ahead of the existing homes.Wipe call — a run's private tree is no longer wiped on the strength of tmux/process liveness alone; the credential signature must be reconciled first.
  • Lab core still never parses a token. Every comparison is signature equality; every copy is bytes-in, bytes-out. The seam's opacity is what lets one rotation policy in core serve every adapter without core learning a single adapter's file layout.
  • A server-side credential proxy remains a live option, and nothing this design does forecloses it — InjectCredentials's one call site, preserved here, is exactly the seam a proxy would replace.