` work, and only a repo that has SSH targets refuses to spawn without a reachable bastion¶

A run can reach an SSH host only by holding a credential for it. The one SSH credential lab manages is the repo's git deploy key, which the vault materializes per operation (ADR-0006) and which the forge itself scopes to one repository. For anything else — a staging box, a build host, a machine the agent is meant to deploy to — the operator's only move today is to hand the agent a key, and from that moment the key is the agent's: readable by any tool it runs, printable into its transcript, sendable anywhere a prompt injection points it. ADR-0067 closed exactly this shape for outbound HTTPS by taking the value out of the run and leaving it the capability; SSH was left as it was.

Warpgate (github.com/warp-tech/warpgate) is an open-source Rust bastion with the same inverted shape for SSH. A target is a host plus the credential Warpgate uses to reach it, stored in Warpgate's own database; a client authenticates to Warpgate as a Warpgate user, selects the target in its SSH username, and Warpgate opens the onward connection with a credential the client never sees, recording the session as it goes. Access is the intersection of the user's roles and the target's roles. One HTTP port (8888 by default) serves both the admin UI and the admin REST API, which lab speaks at <base>/@warpgate/admin/api with an X-Warpgate-Token header; the SSH listener is a separate port (2222 by default).

This ADR records issue #39 in full — one issue, one PR: the configuration surface, an internal client, health, the per-repo identity lifecycle, per-run wiring, and the per-repo target picker. Its one follow-up is #47 (dashboard exposure and a per-run recordings deep link). Every upstream fact below was read off the v0.29.1 source (tag v0.29.1, commit 54f93c8) rather than assumed; there was no live sidecar, which is why everything that mirrors upstream is confined to one file (see Verified wire surface).

The decisions, pinned (grilled with the maintainer on 2026-09-29; the issue's agent brief is the contract, and this list records it rather than reopening it):

Capability, not value, for outbound SSH — the non-HTTP sibling of the credential gateway. A target's real credential, key or password, lives only in Warpgate, entered by the operator in Warpgate's own admin UI. A run authenticates to Warpgate with a lab-issued, run-scoped key that opens nothing but Warpgate, only the targets its repo's role carries, and only until the run is wiped; Warpgate makes the onward connection with the credential the run never held. That is ADR-0067's first pin applied to a second protocol, for the same reason: a key the instance never held is a key no transcript, prompt-injected tool call, or careless cat can leak. The boundary is as deliberate as the model. SSH only: HTTPS is the credential gateway's, and MySQL/PostgreSQL targets are a later issue on this same identity model. Outbound only: Warpgate here is how a run reaches out, never how anyone reaches in — not operator access to the lab host or to instances, and not a front for lab's web UI. Public key only: the repo's Warpgate user never carries Warpgate's password, OTP, or SSO credential kinds.
Generic SSH targets first; git-over-SSH stays on ADR-0006's deploy key. The git credential already has a settled model — encrypted at rest, materialized per operation, one key per repo that the forge itself limits to one repository — and the gap it leaves is small: what a run holds is its own repo's deploy key, which D10 allows by design and which reaches nothing the run was not already entitled to. Arbitrary SSH hosts have no model at all, which is where a bastion buys the most. So the run's git environment is untouched — GIT_SSH_COMMAND keeps pointing at the materialized deploy key — and no Host alias lab writes may match the repo's forge host. That rule is not tidiness: GIT_SSH_COMMAND runs a bare ssh, which on a wired run resolves through PATH to lab's wrapper and so reads the run's own config (below), so a block named like the forge host would reroute git push through the bastion, as a Warpgate user with no target of that name, and break the run's own git. For every host that is not an alias — the forge first among them — that config restores OpenSSH's default lookup, so git's effective ssh configuration is exactly what it was, and the vault's -i/-o options on the command line still win over anything a config file says.
One Warpgate user and one Warpgate role per lab repo — the agent identity's shape, and #35's lifecycle exactly. Warpgate grants access only as the intersection of a user's roles and a target's roles, so "this repo may reach target T" is precisely "T carries the repo's role", and the repo's user holds that role and no other. Both objects are keyed by one slug, warpgate.RepoSlug(repoID), the same derivation as onecli.AgentIdentifier (repo_<32 hex> → repo-<32 hex>), asserted equal by a test so the two sidecars never disagree about what a repo is called; both carry the repo's human name in description, display data lab owns and heals. The lifecycle is #35's word for word: created eagerly when the repo is created; healed at startup and at every target-bearing spawn (a spawn of a repo that has at least one SSH target) as backstops for repos that predate the configuration; a startup pass that is one-way — it creates and renames and never deletes, because Warpgate is shared surface holding the operator's own users and whatever else points at it; both deleted when the repo is deleted; and every one of those writes best-effort with a warning, because the spawn is already the fail-closed point and a bastion outage has no business reaching the repo-create screen. And never delete-and-recreate: the role is the repo's target set and the user holds every live run key, so recreating either to fix a label drops the repo's access and cuts every in-flight run's SSH — a fleet-wide outage traded for a display string.
Access attaches per repo; authentication is per run. Access belongs where every run of a repo needs it, so it hangs on the repo's role once rather than on something minted per spawn. Authentication is a fresh ed25519 keypair per run: lab generates it at spawn, registers the public half as a public-key credential on the repo's user, and hands the private half to that run alone. Revocation is then exact — deleting one credential ends one run's access and leaves every sibling run's key untouched — and the key is worthless outside Warpgate, outside the repo's targets, and after the run. Attribution survives every run of a repo sharing one user: each key's credential label names its run, and Warpgate's per-session log records which key a session authenticated with.
Targets are the operator's, defined in Warpgate; lab only toggles which carry a repo's role. Lab never creates, edits, or reads a target's credential. A target is exactly the object that holds the value this design keeps away from runs, and Warpgate's admin UI already exists to hold it; a lab screen for it would route the credential through lab's API, and binding target CRUD would put a secret-carrying API surface into a client pinned as deliberately partial — the same reasoning that keeps OneCLI's dashboard the only place a secret value is ever typed (ADR-0067). Lab's side is a per-repo SSH targets picker, the analogue of #25's grant picker: it lists Warpgate's SSH targets, shows which carry the repo's role, and a toggle assigns or removes that role on the target through the admin API. Listing is the one Warpgate call that degrades to an error message in the picker rather than failing closed — it is a screen, and a screen that cannot load is not a security event. Toggles touch SSH targets only: the toggle refuses (404) a target id that is not in the SSH listing, so the repo's role can never land on an HTTP or database target that lab does not wire and nobody reviewed for this.
Per-run wiring lands before the claim. When Warpgate is configured for runs and the repo has at least one SSH target, instance.Launch — the single spawn path, after the credential gateway step and before the claim — re-reads the repo's targets from Warpgate, generates the run key and registers its public half, writes the private half into the run's runtime dir, writes the run's ssh_config, a known_hosts pinning Warpgate's host key, and the ssh/scp/sftp wrappers, and prepends the wrappers' directory to the run's PATH. Any failure, Warpgate unreachable included, refuses the spawn with an actionable 400 carrying the cause, with no worktree, no run row, and no claim, so a bastion outage never parks an issue. A repo with no SSH target spawns unchanged — no key, no files, no PATH change, no Warpgate call — and a Warpgate outage never touches it. Two cases reach Warpgate and then find nothing to wire, and both end exactly there: if the fresh read returns no target although the cache said otherwise, the cache is cleared and the spawn proceeds unchanged; if every target it returns is skipped by the alias rule (below), no key is registered, the spawn proceeds unchanged, and one warning names the skipped targets — a key with no alias to use it would be access nobody can exercise and one more thing to revoke. With Warpgate not configured, and on every unwired run, env and argv are byte-identical to before, asserted by a test the way the OneCLI tests assert theirs. A lab with the REST pair set but no --warpgate-ssh-addr is unconfigured for this purpose, not a refusal: it has health and the identity lifecycle, and no address to hand a run.
Fail closed only when the repo has SSH targets — the asymmetry is the decision. A run that has targets and starts without its bastion fails the way ADR-0067 described for HTTPS: minutes in, as auth failures against hosts the agent was told it could reach, spending its budget clock on what looks like a permissions problem and is really a sidecar outage. A run without targets has nothing to lose from the outage, and refusing it anyway would spread an SSH incident onto every repo in the fleet — including every repo that never asked for SSH — for no security gain. The credential gateway fails closed universally because its wiring is universal: all of a wired run's HTTPS goes through the proxy. The bastion's wiring is per target, and so is its refusal.
Addressing is a lab-written OpenSSH client config, handed to ssh by lab's own wrappers, so every ssh-using tool works unchanged. The config has one Host <target-name> block per SSH target: HostName and Port from --warpgate-ssh-addr; User <repo-slug>:<target-name>, Warpgate's target selector (see Verified wire surface); the run key as IdentityFile with IdentitiesOnly yes; and UserKnownHostsFile on the pinned known_hosts. The obvious placement, .ssh/config in the instance HOME, does not work, and that was verified rather than assumed: OpenSSH (10.5p1, the version checked) locates the per-user config through the passwd entry's home directory, not $HOME, so a host run reads the lab service user's /var/lib/lab/.ssh/config and a --userns=keep-id container reads nothing, its generated passwd entry having home /; and ssh -o Include=… is refused outright ("Include directive not supported as a command-line option"). So the config lives in the run's runtime dir, and a wired run gets ssh, scp, and sftp wrappers in a per-run bin dir PREPENDED to its PATH, each exec'ing the real binary — the first one on PATH outside the wrapper's own dir, so it works with the host's ssh and with whatever a dev image ships — with -F <config> ahead of the caller's arguments. scp and sftp need wrappers of their own because they launch ssh by a compiled-in absolute path rather than through PATH; rsync -e ssh, git's GIT_SSH_COMMAND, ansible, and anything else that runs ssh by name resolve to the ssh wrapper. -F replaces the per-user and the system config both, so the run's config ends by restoring them for every host that is not an alias (see the shape below): the forge, and everything else, resolves exactly as it did without the wrapper. ~/.ssh/config in the instance HOME is a symlink to the same file, for discoverability and for libraries that read $HOME/.ssh/config themselves (paramiko and its kind). ssh <target-name>, scp, sftp, and rsync -e ssh then work with no flags and nothing for the agent to learn. Configuration is the one layer every such tool shares; the alternative — teaching the agent -F/-i/-p/-o flags in its context file — works until the first tool it drives builds its own ssh command line.
Warpgate's host key is trusted by configuration: --warpgate-ssh-host-key names the key runs trust, and left unset, lab accepts whatever the listener presents. With the setting present, every run's known_hosts is rendered from the configured key(s) verbatim and checked strictly, and every target-bearing spawn scans the listener first: if it presents none of the configured keys, health reports degraded with both sides named and the spawn fails closed until the setting is changed — a man in the middle between lab and the bastion can block spawns, never widen what a run trusts. With the setting absent, the scan that wires a run is what that run trusts: every key the listener presents goes into its known_hosts. That is trust on each spawn, taken by lab on the operator's host, and it is the operator's explicit choice rather than a default lab took for them: health and the startup log list the observed fingerprints so the setting can be filled in from them, and the ops guide says to once the sidecar is up. accept-new inside a run is explicitly rejected even so, because inside a run it is silent and unobservable: every run starts with an empty HOME, so every run would be a first use, and whatever answered on the bastion's address at that moment would be trusted with nothing logged and nothing for health to show. Lab's spawn-time scan trusts the same thing when nothing is configured, but it does so once per run, before the claim, in a place lab logs and health reports, and the known_hosts it writes holds that decision still for the run's lifetime. Upstream offers no alternative to scanning, either: Warpgate 0.29.1 keeps its host keys in its database and exposes them through no API. ADR-0006's accept-new for the forge host is a different known_hosts and stays exactly as decided — with one addition that keeps it from reaching the bastion: git's GIT_SSH_COMMAND passes that known_hosts and accept-new as -o options, which beat the run's config, so a git clone staging:app.git on a wired run would reach Warpgate through the wrapper and trust its key on first use. Lab therefore also appends the run's bastion known_hosts lines to its vault known_hosts (after what was seeded into it, never replacing it), where accept-new finds the key and refuses a changed one.
Five optional settings, entirely off when unset, on the OneCLI pattern and its pairing rule. --warpgate-url / LAB_WARPGATE_URL / services.lab.warpgate.url is the admin API base lab itself dials, typically loopback; unset turns the whole integration off. --warpgate-admin-token-file / LAB_WARPGATE_ADMIN_TOKEN_FILE / .adminTokenFile is a file holding the admin token, checked under the same 0600 permission rule as the master key file and the OneCLI key file and never auto-generated, because Warpgate mints it. Those two are a pair enforced at Parse, because half a REST config is dead config. --warpgate-ssh-addr / LAB_WARPGATE_SSH_ADDR / .sshAddr is the host:port a run dials, independently settable and outside the pairing for exactly --onecli-gateway-url's reason: the container runner pins host.containers.internal and host.docker.internal to 127.0.0.1 inside the container (ADR-0052 as amended by #216), so a containerized run cannot reach the host's loopback and must be handed an address routable from its netns. --warpgate-ca-file / LAB_WARPGATE_CA_FILE / .caFile is the PEM lab trusts for the admin API, since Warpgate ships a self-signed certificate by default; unset means the system roots. --warpgate-ssh-host-key / LAB_WARPGATE_SSH_HOST_KEY / .sshHostKey is the trusted public host key of the SSH listener (authorized_keys or ssh-keyscan form, several separated by newlines or commas), parsed and canonicalized at Parse, refused without --warpgate-ssh-addr because a key with no listener to vouch for is dead config, and optional: unset, runs trust whatever the listener presents (decision 4). A run is wired only when URL, token file, and SSH address are all set. The NixOS module emits each flag only when its option is non-null, as the onecli.* options do.
The client binding is deliberately partial, and one file mirrors upstream. internal/warpgate binds health; ensure, find, and delete of a repo's user and role; role assignment on a user and on a target; SSH-target listing, all and per role; and public-key credential add, list, and remove — nothing speculative, for ADR-0067's reason that every bound endpoint is one more thing an upstream release can break. internal/warpgate/wire.go holds every path and every struct that mirrors Warpgate's JSON, so a correction against a real build is one file plus its tests. Tests run against a stubbed admin API, never a live sidecar.
Health beside the credential gateway's, on the same contract. GET /api/v1/warpgate/health always answers 200, the state is the payload — off, ok, degraded, or unreachable — and it folds three components: the admin API (reachable, and whether the token was accepted as admin), the SSH listener, and the host-key state (unpinned, pinned, mismatch, unreachable). Nothing configured is off; every configured component unreachable is unreachable; any one unreachable, a token not accepted as admin, or a host-key mismatch (or a first pin that failed to store) is degraded; anything else is ok. The web UI shows it next to the credential gateway status. An unconfigured lab is not an unhealthy lab, as in ADR-0067.
The operator runs upstream's container; lab ships no deployment for it. docs/ops.md gains a section beside the OneCLI one: run upstream's image and its setup to obtain the admin credentials, bind the HTTP port (admin UI and API) to loopback, bind the SSH port on an address container runs can route to — the same deliberate exception to the guide's bind-to-loopback rule that the credential gateway's proxy port already is — create the admin token file, and reach the admin UI over an SSH tunnel. No NixOS option deploys Warpgate, exactly as none deploys OneCLI: it is a separately released system with its own database, encryption, and upgrade path, which lab consumes and does not own.
Dashboard exposure is deferred to #47, and its shape is already expected. Warpgate's admin UI shares the admin API's port, so exposing it through lab raises the questions ADR-0067 settled for OneCLI's dashboard — which origin, gated by lab's session how, with what cookie scope — and #47 is expected to answer them with the same configurable off / port / subdomain shape. Those are operator convenience and auth plumbing, not whether a run can reach a host, so they do not belong in the slice that makes runs work. Until #47, off is the only mode and the SSH tunnel is the access path, which costs nothing because the admin port is loopback-bound anyway.
Session recording is orthogonal. Warpgate can record every session it proxies (its setup enables that with --record-sessions), and that stays Warpgate's: lab neither reads, stores, nor replays a recording. Linking a run to its recordings is #47's.

The implementation pins, settled on the same issue while fixing the interfaces the parallel work builds against:

A lab-side cache of each repo's SSH targets, repo_ssh_targets (migration 0023). The fail-closed rule must be decidable while Warpgate is down: whether a repo has any target decides whether the spawn calls Warpgate at all, so answering that by calling Warpgate would collapse the rule's two halves — with Warpgate down, lab would have to refuse every spawn or admit every one. The table holds (repo_id, target_id, target_name) and nothing else. The picker's toggle writes one row as it assigns or unassigns; every full read of the truth — the picker listing, the startup reconcile, and every target-bearing spawn — REPLACES the repo's whole cached set. An empty cache means the spawn makes no Warpgate call, and the aliases a run gets always come from that spawn's fresh read, never from the cache.
The cache does not contradict ADR-0067's "no parallel lab-side assignment model". That pin rejects a second source of truth that lab must keep in step and reconcile after dashboard edits. This table is never authoritative: no access decision is made from it — Warpgate enforces the role intersection on every connection regardless of what lab believes — it is overwritten wholesale by every read rather than merged, and the only question it answers is whether a spawn should ask Warpgate at all. It can be stale in one direction only (see Consequences); a target removed in Warpgate's UI costs nothing, because the spawn re-reads before writing any alias.
Matching rules. The Warpgate username and the role name are both the slug. Users match by exact username, because Warpgate's user search is a case-insensitive substring match. Roles have no uniqueness constraint upstream, so lab matches the exact name and, when duplicates exist, takes the lexicographically smallest id — deterministically, so every call site resolves the same role. is_default is always false: a default role is auto-granted to every new Warpgate user, which would hand this repo's targets to every future repo and every user the operator ever creates. The display-name heal GETs and then PUTs the whole user, because Warpgate's PUT replaces the object and resets any field left out; only description ever moves.
The run key's label carries the run ID, because its comment cannot. Warpgate strips the comment from every public key it registers, so the brief's "run ID in the key comment" cannot survive registration. The run ID rides in the credential's required label instead — lab-run:<runID>, with warpgate.RunKeyLabel / warpgate.RunIDFromLabel the only encoder and decoder — and the OpenSSH comment is set to the same string as well (harmless, and it survives if a future Warpgate keeps it). Every sweep touches only keys whose label parses as a lab run label, so a key an operator added to a repo user by hand is never removed: #35's rule against deleting what lab did not create, one level down.
Where the trust lives. In configuration, not in lab's database: --warpgate-ssh-host-key holds the trusted public key(s), and Parse stores it canonical — authorized_keys lines, <type> <base64>, sorted and de-duplicated — whatever form it was given in (a comment after the key, or the [host]:port prefix of an ssh-keyscan line, is dropped; a fingerprint or a private key is refused with the entry named). Nothing about the host key is in a settings row, so nothing about it is reachable through the generic settings PATCH or changed by restoring a database backup, and the NixOS module carries it like every other setting. An earlier revision of this implementation kept a trust-on-first-use pin in a settings row with an accept endpoint; the maintainer replaced it with the setting before merge (see the superseded option below).
How the host key is checked. Lab scans once per host-key algorithm (ssh-ed25519, rsa-sha2-512, ecdsa-sha2-nistp256/384/521) and keeps every distinct key; Warpgate 0.29.1 serves an ed25519 and an RSA key. With trusted keys configured, the listener must present at least one of them — compared by marshaled key — and the run's known_hosts holds every configured key: OpenSSH prefers the host-key algorithms it already has a key for, so one configured key is enough and the run negotiates one lab trusts. With none configured, the run's known_hosts holds every key the scan observed. The host pattern is the current --warpgate-ssh-addr ([host]:port; Parse stores the address canonical — 10.88.0.1:02222 becomes 10.88.0.1:2222, a signed port is refused — because ssh looks the bastion up by the port it parsed, and a pattern spelled otherwise never matches), so an address change needs no new key: a new address serving a configured key matches, a different server mismatches. Every target-bearing spawn scans; that doubles as the SSH listener's reachability probe, and it catches a changed key as a legible refusal at the spawn rather than a host-key-changed abort inside the run.
The admin token: Warpgate's static admin token is recommended. warpgate run --enable-admin-token with WARPGATE_ADMIN_TOKEN in the environment never expires; the image's default command does not pass the flag, so the ops guide has the operator override it. A per-user API token with admin permissions also works but carries an expiry the operator must track, and an expired token surfaces as degraded health and refused target-bearing spawns rather than as silence. The token is sent as X-Warpgate-Token, never as a Bearer token, read once at startup, and never logged.
TLS. With --warpgate-ca-file set, lab trusts only that PEM — not the system roots besides; the admin API is one known endpoint and the system roots would widen trust for nothing — and Go's verifier accepts a self-signed leaf in the pool as its own anchor. Warpgate's generated certificate carries only the SANs localhost and warpgate.local, so the URL must name https://localhost:8888; https://127.0.0.1:8888 fails verification against that certificate, which has no IP SAN. Unset, the system roots apply, which fits a Warpgate fronted by a publicly issued certificate.
The files a wired run gets. In the runtime dir (homes.RuntimePath(runID), bind-mounted host-identically into container runs): warpgate-run-key, the ed25519 private key, 0600 — OpenSSH refuses a private key others can read; warpgate-known-hosts, 0600; warpgate-ssh-config, 0600, the run's OpenSSH client config; warpgate-bin/ (0700) holding the three wrappers ssh, scp, and sftp (0700) — POSIX sh generated per run with the absolute paths baked in single-quoted, each scanning PATH for the real binary while skipping its own dir and exiting 127 with a message when there is none. The pinned lines are also appended to the run's vault known_hosts in the same dir (see the host-key pin above). One level up, at the root of the per-run instance dir (<instances>/<runID>/, the parent of home/, runtime/, and imports/): warpgate-run-key.json, a 0600 marker {"user_id":"…","key_id":"…"} carrying no secret, written immediately after the key is registered so every wipe path can revoke it. It is the one wiring file kept out of the runtime dir because that dir is bind-mounted read-write into container runs, and a marker the run can delete or rewrite is a revocation the run can defeat — its key would outlive it; the instance dir root is never mounted into a container, and the wipe removes it whole, marker included. A host run is no isolation boundary for it — its agent is the lab user — and nothing further is attempted there; the startup sweep is the backstop either way. In the instance HOME: .ssh/ (0700) with .ssh/config a symlink to <runtime>/warpgate-ssh-config by absolute host path, which resolves inside a container too because the runtime dir is mounted host-identically. The config and the wrappers name only absolute runtime-dir paths, which resolve identically on both sides of the container boundary; the HOME does not, since inside a container it is /home/agent (ADR-0052). PATH: a wired host run's spawn env gets PATH=<runtime>/warpgate-bin: followed by lab's own PATH; a wired container run gets the same prefix on podmanx.PATH — exactly one PATH in the container env, and the host's PATH is never forwarded inward. An unwired run's env and argv are byte-identical: no PATH entry on a host run, podmanx.PATH unchanged in a container.
Revocation goes through the one pre-wipe hook. instance.Service.RevokeBastionKey(ctx, runID) reads the marker from the root of the run's instance dir and deletes that one credential — a 404 is success, any other failure a warning — and cmd/lab composes it into the single pre-wipe hook on instancehome.Manager, beside credrotate's adopt-check (ADR-0055). Stop, the AFK reaper, launch rollback, and the orphan sweep therefore all revoke with no per-site code; the hook exists precisely because every path that destroys a run's tree needs one last look at it first. A registration that fails ambiguously — Warpgate committed the POST, the client saw a timeout, a cancelled request, or an unreadable answer — would leave a key no marker names, so the refused spawn first removes, best-effort, every key labelled for its run. instance.Service.SweepBastionKeys(ctx) runs at startup and catches what the marker cannot — a crash between registration and the marker write, or a wipe whose revoke call failed: for each repo's Warpgate user it removes keys whose label names a run that is neither active in the store nor still on disk. A run whose instance dir exists is live or in flight — a spawn racing the sweep has its tree and its committed key before its run row, and for an instant before its marker — or an orphan tree that instancehome's own sweeps will wipe, revoking its key through the hook; once the tree is gone the next startup sweep takes the key. It is bounded and stops at the first sidecar-level error with a single warning, like #35's startup pass, and cmd/lab retries a pass cut short with backoff (1, 2, 4, 8, then every 15 minutes) until one completes, since a Warpgate still starting beside lab is the common reason a pass fails. A run alive across a lab restart keeps its key.
What the seeder is given. A wired run's context file gets a short SSH targets section: the aliases, how to use them (ssh <alias>, scp, rsync -e ssh), that no credential for them is in the instance, and that unlisted hosts are not reachable through lab. The seeder is handed alias names and nothing else — no address, no username, no key path — so nothing credential-shaped reaches the render, which is ADR-0067's rule for granted service names. An unwired run renders no section, byte-identical to before.
The ssh_config shape is below — one block per target, in deterministic order by alias, under a header saying the file is rewritten at every spawn, then the restoration of OpenSSH's default lookup for every other host.

# Generated by lab for run <runID>: SSH targets reached through the Warpgate bastion (ADR-0068).
# Rewritten at every spawn; edits are lost.

Host staging
  HostName 10.88.0.1
  Port 2222
  User repo-0123…:staging
  IdentityFile "/…/runtime/warpgate-run-key"
  IdentitiesOnly yes
  UserKnownHostsFile "/…/runtime/warpgate-known-hosts"
  GlobalKnownHostsFile /dev/null
  StrictHostKeyChecking yes
  UpdateHostKeys no
  PasswordAuthentication no
  KbdInteractiveAuthentication no

# -F (passed by this run's ssh/scp/sftp wrappers) replaces OpenSSH's default per-user and system
# config lookup; restore it for every other host, the forge included.
Match all
Include "/var/lib/lab/.ssh/config"
Include /etc/ssh/ssh_config

Each of the last five directives in a target block closes one door: no system file can vouch for the bastion, nothing outside the pin is ever accepted, the server cannot push keys into the pinned file, and an auth failure fails fast instead of prompting in a pane nobody watches. The trailing Match all gives back exactly what -F took away, and Match all itself is load-bearing: an Include placed after a Host block would otherwise be scoped to that block. The per-user Include is the lab service user's own config — its passwd home, resolved with os/user, plus /.ssh/config, the very file a host run's ssh read before — and it is emitted for host-runner runs only, since inside a keep-id container the passwd home is / and ssh never read a per-user config there. It is omitted when that home is unknown or when the path resolves, symlinks followed, to the run's own config: a self-Include is a hard OpenSSH error ("Too many recursive configuration includes"), while an Include of a missing file is a silent no-op (both verified). A target becomes an alias only if its name matches ^[A-Za-z0-9][A-Za-z0-9._-]*$ — no #, %, whitespace, or glob characters — and does not equal, case-insensitively, the repo's forge host; a skipped target is a warning, never an error. There is no Host * block and no directive of lab's that applies to every host, ever — the trailing Match all carries only the two Includes, because a wildcard setting would apply the bastion's settings to every host the run dials, the forge included — and accept-new is never written.

Verified wire surface (Warpgate 0.29.1)¶

Read off upstream source at v0.29.1; internal/warpgate/wire.go is the single point of correction if a live build disagrees.

Authentication and roots. Every admin request carries X-Warpgate-Token: <token> — there is no Bearer scheme. A missing or rejected token is 401 with an empty body; an admin token lacking a permission is 403. The admin API root is <--warpgate-url>/@warpgate/admin/api. The only health probe is GET <--warpgate-url>/@warpgate/api/info, on the gateway API rather than under /admin/api: it answers 200 unauthenticated, and with an accepted admin token its version and admin_permissions become non-null. Error bodies have no JSON envelope — a declared 4xx is a bare JSON string or empty, an internal error is text/plain — and lab sends Accept: application/json, because text/html gets an HTML page.

Operation	Method and path (under the admin root)	What lab relies on
Find user	`GET /users?search=<slug>`	Case-insensitive substring match — lab filters for the exact username
Create user	`POST /users` `{username, description}`	201; a duplicate username is 400 with body `"username"`, not 409
Heal user	`GET` then `PUT /users/{id}`	PUT replaces the whole object; omitted fields reset, so lab always sends the full user
Delete user	`DELETE /users/{id}`	204; 404 when absent
Find role	`GET /roles?search=<slug>`	Duplicate names are allowed — lab takes the exact name, smallest id
Create role	`POST /roles` `{name, description, is_default: false}`	201
Heal / delete role	`PUT` / `DELETE /role/{id}`	Singular `/role/`; only list and create are `/roles`
Grant role to user	`POST /users/{id}/roles/{role_id}`, body `{}`	Needs `Content-Type: application/json` (415 without). 201 even when already granted; a 409 here means an unknown role, never "already granted"
User's roles	`GET /users/{id}/roles`	Lists revoked and expired rows too — filter on `is_active`
List targets	`GET /targets`	Lab keeps `options.kind == "Ssh"` only; `allow_roles` is always `[]` here
Role's targets	`GET /role/{id}/targets`	One call, full targets of that role — how lab reads a repo's set
Assign role to target	`POST /targets/{id}/roles/{role_id}`	201; 409 when already assigned, which lab treats as success
Unassign role from target	`DELETE /targets/{id}/roles/{role_id}`	204; a hard delete, so a repeat is 404, which lab treats as success
List user keys	`GET /users/{id}/credentials/public-keys`	`[{id, label, date_added, last_used, openssh_public_key}]`
Add user key	`POST /users/{id}/credentials/public-keys` `{label, openssh_public_key}`	201; the key's comment is stripped, so the run ID rides in `label`; a malformed key is a 500 (lab generates its own)
Remove user key	`DELETE /users/{id}/credentials/public-keys/{id}`	204; 404 when absent, which lab treats as success

Target bodies carry credentials. GET /targets and GET /role/{id}/targets return each target's stored secrets — passwords, private keys, tokens — in plaintext or in an encrypted envelope. Lab's wire structs therefore decode only id, name, description, and options.kind, and a targets response body is never folded into an error or a log line: a debugging aid must not become the leak this design exists to close.

The target selector is the SSH username, user:target, username first. Two edge cases: if a # appears anywhere in the selector it becomes the separator instead of :, and a selector beginning with ticket- is parsed as ticket authentication. Lab's slug, repo-<32 hex>, contains neither : nor # and never begins with ticket-, and the alias rule keeps # (and %, which would also meet ssh_config's own token expansion in the User line) out of target names. A target the user's roles do not reach fails authentication; an empty target would get Warpgate's interactive menu, which lab's config never produces.

Host keys and deployment. No API exposes the SSH host keys; Warpgate stores an ed25519 and an RSA key in its database, so a scan is the only way to learn them. Default ports are 8888 for HTTP (admin UI and API) and 2222 for SSH, which is off unless ssh.enable is set. The image is ghcr.io/warp-tech/warpgate:0.29.1; unattended-setup produces the admin credentials, and run --enable-admin-token with WARPGATE_ADMIN_TOKEN enables the static token. MFA enforcement is a global Warpgate parameter, and requiring a second factor there breaks key-only SSH for the repo users.

Not verified and not relied on: whether deleting a user cascades its public keys (lab revokes keys itself), the label length limit (lab-run:<runID> is short), and the status for an unknown id on target-role assignment (lab only assigns ids it has just listed).

Status¶

Accepted. Settled via issue #39 (2026-09-29), grilled with the maintainer; the agent brief on that issue is the contract, and the implementation pins above were settled on the same issue as its interfaces were fixed. Follow-up: #47 (dashboard exposure modes and a per-run recordings deep link). The non-HTTP sibling of ADR-0067 — whose status line now points here as its SSH counterpart — built on that ADR's house pattern: off-by-default settings with a paired REST half and an independently settable run-facing address, a partial client with one wire file, the always-200 health contract, a per-repo identity with #35's lifecycle, a pre-claim fail-closed step in instance.Launch, and the seeder's names-only rule. Leaves ADR-0006 untouched: the vault, the master key, per-op materialization, the two-slot split, and git's GIT_SSH_COMMAND with its own known_hosts and accept-new — a wired run's per-run copy of that known_hosts only gains the bastion's pinned lines. Reuses ADR-0055's pre-wipe hook and sweep as the revocation path, a second consumer of a hook built for one last look before a run's tree is gone. Extends ADR-0052 without changing it: the run key, known_hosts, ssh config, and wrappers ride the runtime dir's existing host-identical bind mount, the only container-side difference on a wired run is a prefixed PATH (still exactly one), and its host.containers.internal pin is why --warpgate-ssh-addr is its own setting.

Considered options¶

Route git-over-SSH through Warpgate first. Rejected: ADR-0006's deploy key already has a settled model, and the gap it leaves is small — the run holds its own repo's key, which the forge already limits to that repo. Moving git behind the bastion would put the one path every run depends on — clone, fetch, push, and the PR that is its done-signal — behind a new sidecar, turn a Warpgate outage into a git outage for every repo, and give the deploy key a second home beside the vault. The largest blast radius on this list, for the smallest gain.
A Warpgate user per run. Rejected: Warpgate access is a role intersection, so each run's user would need the repo's role granted at spawn and revoked at stop — two lifecycle writes instead of one key — and a crash would orphan a whole user holding live access rather than one key on a user. Warpgate's user list would churn with a row per run ever spawned. The one thing it buys, per-run attribution, the per-run key already gives.
SSH certificates from a lab CA instead of registered keys. Rejected: Warpgate 0.29.1's SSH listener does not accept certificate credentials at all — its certificate credential kind is explicitly unsupported for the SSH protocol. Even if it did, lab would gain a signing key whose compromise mints access to every repo's targets, and exact revocation — delete one credential — would give way to expiry windows or a revocation list.
StrictHostKeyChecking accept-new inside runs. Rejected: every run starts with a fresh HOME and an empty known_hosts, so accept-new makes every run a first use, taken silently inside a pane nobody watches. Anything answering on the bastion's address when a run first connects is trusted, and the agent then sends its commands and files through it, with nothing logged and nothing for health to report. Lab's own spawn-time scan, when no trusted key is configured, trusts the same listener — but before the claim, on the operator's host, logged, reported by health with the fingerprints, and written into a known_hosts the run checks strictly for its whole lifetime; and it is one setting away from a real pin.
A trust-on-first-use pin in lab's database, with an accept endpoint. The implementation first shipped this: lab's first successful scan became the pin in a settings row, a later mismatch blocked target-bearing spawns until an operator accepted one of the observed fingerprints through POST /api/v1/warpgate/host-key/accept (and a button on the status card). Superseded before merge, on the maintainer's call: the trust then lived in state rather than configuration — taken by whatever answered at lab's first start, invisible to the NixOS module, and restored along with a database backup — and the accept pinned the listener's whole current key set on the strength of one verified fingerprint, which let a key answered on another algorithm ride in beside the one inspected. A configured key is inspectable, versioned with the deployment, and exactly what the operator verified; leaving it unset is an explicit choice to accept what the listener presents, made visible rather than taken once and forgotten.
The ssh config in the instance HOME alone, relying on $HOME. Rejected: it is invisible to OpenSSH, which locates ~/.ssh/config through the passwd entry — the lab service user's home on a host run, / in a keep-id container — so ssh <target> would silently fall through to a DNS lookup of the alias name. Verified on OpenSSH 10.5p1 before the decision was taken. The HOME keeps a symlink to the real file, for the tools that do read $HOME.
podman --passwd-entry so a container's passwd home is the instance HOME. Rejected: it fixes container runs only. Host runs share the lab service user's passwd entry and would still need the wrapper, and one uniform mechanism across both runners beats two that have to be reasoned about separately.
An Include in the service user's real ~/.ssh/config or in /etc/ssh. Rejected: it mutates host-level files shared with lab itself and with every other run, and it is not per-run — one run's aliases and key path would be visible to every concurrent run and to lab's own server-side git operations, and a crash would leave the edit behind.
Lab creating Warpgate targets. Rejected: a target is its credential, so a lab form for it carries the very value this design keeps out of lab and out of runs through lab's API, and binding target CRUD adds a large, secret-carrying surface to a deliberately partial client — for a job Warpgate's admin UI already does. The same reasoning leaves OneCLI's dashboard the only place a secret value is typed.
Fail closed on every spawn whenever Warpgate is configured. Rejected: it turns an SSH outage into a fleet outage, stopping repos that have no SSH target and never asked for one, for no security gain — a run without targets has no bastion access to lose. Universal fail-closed is right for the credential gateway only because its wiring is universal.
No lab-side cache: ask Warpgate at every spawn. Rejected: with Warpgate down, lab could not tell whether a repo has targets, so the fail-closed rule becomes undecidable — refuse every spawn (the previous option, by the back door) or admit every spawn (fail open for exactly the repos the rule protects). It would also put a Warpgate round trip on every spawn of a repo without targets, which the brief forbids.
The run ID in the key's comment, as the brief first put it. Rejected on upstream fact: Warpgate strips the comment at registration. The required label survives it and carries the run ID instead.
Fetching the host key from Warpgate's API. The route does not exist in 0.29.1; a scan is the only way to learn the key, and the operator reads it on the Warpgate host itself to configure it. (Configuring a single key type is fine, and the ops guide says so: OpenSSH prefers the host-key algorithms it has a known key for, so a run with one trusted ed25519 key negotiates ed25519.)

Consequences¶

A run reaches operator-defined SSH hosts holding no credential for any of them. What it does hold, the run key, opens only Warpgate, only its repo's targets, and only until its wipe — and every session passes through Warpgate's own audit and recording, which lab gains without building any of it.
Five new optional settings, env vars, and NixOS options — --warpgate-url, --warpgate-admin-token-file, --warpgate-ssh-addr, --warpgate-ca-file, --warpgate-ssh-host-key, and services.lab.warpgate.{url,adminTokenFile,sshAddr,caFile,sshHostKey} — the first two enforced as a pair at Parse, the host key parsed there and refused without the address, the other two validated as shapes only.
A new internal/warpgate package whose wire.go is the single point of correction, with host-key helpers (scan, format, compare, render known_hosts) on golang.org/x/crypto/ssh.
New persisted state: the repo_ssh_targets table (migration 0023, both dialects; the cache, cascading with the repo). The host key is configuration, not state.
New authenticated HTTP surface: GET /api/v1/warpgate/health and GET /api/v1/repos/{id}/warpgate/targets with PUT/DELETE /api/v1/repos/{id}/warpgate/targets/{targetId}. The web UI gains the per-repo SSH targets picker and a bastion status next to the credential gateway status.
docs/ops.md gains a sidecar section that names the SSH port as the second deliberate exception to binding co-located services to loopback, beside the credential gateway's proxy port, so the guide does not contradict itself.
CONTEXT.md gains SSH bastion, SSH target, and run key, and its flagged ambiguities record that "user", "role", and "target" mean Warpgate's objects in this area and something else everywhere else in lab.
A new sidecar to operate — the largest genuine cost, and ADR-0067's again. A second process with its own database, encryption, backups, and release cadence, and an admin credential to it that lab holds: with the recommended static token, a compromise of lab's host is full admin over the bastion. That is no worse than today's alternative of keys handed straight to agents, but it is stated. Lab stays self-contained for the deployment that leaves the settings unset.
Every target-bearing spawn pays for the bastion: one SSH handshake per scanned host-key algorithm, plus a few REST calls — the identity ensure, the role's targets, the key registration — and one more to revoke at stop. Spawns of repos without targets pay nothing.
The cache can lag an assignment made in Warpgate's UI. A target given a repo's role directly in Warpgate is not wired until lab next reads that repo's truth — at startup, when the picker is opened, or at a target-bearing spawn — and for a repo whose cache is empty the spawn read never happens, so such an assignment waits for the next picker visit or restart. Assigning from lab's picker has no lag.
Keys orphaned by a crash stay valid until the next startup sweep or wipe. Their private halves sit in runtime dirs only lab's uid can read, and they authenticate only as their repo's user, but until then they are live. A startup sweep that finds Warpgate down is retried until it completes; a key whose run's tree was still on disk at that sweep waits for the one after the tree is gone.
The system ssh config becomes subject to OpenSSH's owner/mode check on every wired run. The run's config pulls /etc/ssh/ssh_config in by Include, and OpenSSH refuses an included file owned by anyone but root or the user, or writable by group or others ("Bad owner or permissions", fatal), a check its own default read of that file skips — so a dev image with a loose /etc/ssh/ssh_config breaks every ssh in a wired container run, git included, where it worked unwired.
Global MFA enforcement in Warpgate breaks key-only SSH. Requiring a second factor in Warpgate's mfa_enforcement parameter makes every repo user's key login insufficient, and target-bearing runs then fail authentication inside the run, where the spawn cannot catch it. The ops guide says so.
Per-run attribution is keyed on the key, not on its label. Warpgate's session log records which key a session used; the label maps that key to a run only while the credential exists, and revocation deletes it. Recovering the run behind a session after the run is gone needs the key's fingerprint on lab's side, which #47's per-run recordings link has to settle.
A wired run's PATH carries a per-run wrapper dir, and the aliases exist only for tools that go through it. which ssh in a wired run names lab's wrapper, and a host run's PATH is now composed by lab rather than inherited. A tool that execs ssh by absolute path without reading ~/.ssh/config itself bypasses both the wrapper and the symlink and sees no aliases; calling it through ssh by name, or passing -F, is the fix. Unwired runs carry no wrapper and no PATH change.
A target whose name is not alias-safe is unreachable by alias — one containing #, whitespace, or a glob character, or equal to the forge host. Lab warns and skips it, and a repo whose targets are all skipped spawns unwired; renaming the target in Warpgate is the fix.
Lab owns the display strings on the repo's Warpgate user and role. A description edited in Warpgate's UI is stomped at lab's next ensure, exactly as #35 decided for OneCLI's agent names.