Skip to content

The definition of done for an AgentProvider adapter becomes a two-tier conformance bar: one CI-run providertest suite, plus four required live spikes evidenced by a committed compat record

Each provider-seam ADR so far pinned its own obligation directly onto SeedMeta or the seam and proved it with a hand-written test living inside internal/provider/claudecode: ADR-0033 declared the attribution-marker dialect and its dual enforcement points, ADR-0034 pinned the login-session prefix rule and the generic host-config maps, ADR-0035 pinned the lab-owned .local context-file name. Every one of those tests exercises claude-code specifically. Nothing runs the full set against a candidate adapter — a second implementation (#2, Codex) could satisfy the interface, compile, and still silently skip an obligation that only ever got checked because claude-code's own author happened to remember it. "Passes review" is not an executable bar, and the gap only grows as providers accumulate.

Not every obligation can be made CI-executable, though. Whether an adapter's transcript survives a context clear, whether its reply/dialog/interrupt keystroke recipes are safe against the real TUI, whether the CLI actually reads the lab-owned context file, and what attribution markers the CLI actually writes into a commit — all of that requires a live binary and a human watching it, exactly the boundary ADR-0008 already drew around internal/compat's live-verified pins. A conformance bar that pretended otherwise would either fake the live obligations (worthless) or drag real credentials and a real CLI into CI (rejected by ADR-0008 already). So the bar splits into two tiers matching that boundary: Tier 1 is fully automatic, product-path assertions runnable in CI against a hermetically-constructed adapter; Tier 2 is the live-verification work ADR-0008 already implied per fragile coupling, now enumerated as four named spikes and made a mandatory, evidenced gate rather than an unstated expectation.

The pins, decided:

  • Two tiers, one bar. "Passes the conformance suite + checklist" is the executable definition of done for any AgentProvider adapter. Tier 1 runs in CI; Tier 2 does not; both are mandatory before an adapter ships.
  • Tier 1 is one entry point, not per-adapter assertions. providertest.Conformance(t, p, fx) (internal/provider/providertest/conformance.go) runs the same ten obligations against every adapter. A new adapter is one entry in the registration table (internal/provider/conformance_test.go, TestConformance, package provider_test) constructing itself hermetically (temp dirs, a fake runner) plus a providertest.Fixture carrying its own real attribution-marker samples as ground-truth vectors — not synthetic ones.
  • The marker check runs both real engines, not one. An RE2-only check would happily pass a pattern like (foo|bar) that the pre-push hook's actual engine — POSIX BRE grep -i — reads as the literal string (foo|bar), not an alternation. scrub-markers therefore compiles each AttributionSample's match via provider.CompileScrubPatterns (RE2) AND shells to a real grep -i (BRE, skipped when absent), plus the union path provider.NewRegistry(p).ScrubRegexps() — the exact compiled objects the agent-API body sanitizer runs. This is what makes the BRE∩RE2 common-dialect rule (ADR-0033) executable instead of aspirational: the product's own compilers do the checking, not a description of them.
  • Seeding checks run the real seeder, not a model of it. seeding-exclude-coverage and seeding-incogni spin up a real temp linked git worktree and run the provider's own SeedWorkspace plus the real internal/seeder in production order. git status --porcelain coming back empty is the executable form of "ExcludeEntries cover every seeded path" — no hand-maintained list of seeded paths to keep in sync with what the seeder actually writes.
  • Broken adapters are private to the suite's own tests. Deliberately-broken fake adapters prove each obligation actually fails when violated, and the failure names the obligation it broke, not a generic assertion diff — so a future obligation added to the suite is provably load-bearing, not decorative.
  • Tier 2 stays required and out of CI, evidenced by a committed artifact. ADR-0008 already draws this line: no real CLI, no credentials, in CI. The four spikes — transcript identity across a context clear, reply/dialog/interrupt hazard-checked against the real TUI, context-file discovery (the CLI verifiably reads the lab-owned name), and incogni attribution ground truth (what the CLI actually writes into commits/PRs, reconciled against the adapter's ScrubPatterns) — are live-run by a human on a host with the real CLI and recorded at internal/compat/<providerID>/compat.md, following claude-code's existing internal/compat/compat.md shape. Claude-code's record stays exactly where it is, as the worked example.
  • Four hard requirements carry no capability-omission escape hatch. A durable, locatable transcript; the lab-owned .local context-file name (never satisfied by declaring a tracked one); live verification before merge; and core staying adapter-agnostic (internal/provider/neutrality_test.go, web/src/providerNeutral.test.ts). Every other seam capability is optional-by-type-assertion (ADR-0017); these four are not.

Status

Accepted. Pins issue #80 as the two-tier conformance bar for every AgentProvider adapter. Extends ADR-0008's live-verification bar (Tier 2's four spikes are exactly the per-coupling live verification ADR-0008 already required, now enumerated and made a named merge gate instead of an implicit expectation); enforces ADR-0033's BRE∩RE2 common dialect and union scrub-marker path executably (patterns-dialect, scrub-markers); enforces ADR-0034's login-session prefix rule (login-session); enforces ADR-0035's lab-owned context-file rule (context-file-lab-owned); and conforms adapters against ADR-0026's seam — the AgentProvider interface this suite exists to hold every implementation to. Relates to #2 (Codex): it adopts this bar as its own definition of done, the first adapter proved against it after claude-code.

Amended by issue #124 (2026-07-10): the Tier-2 checklist gains a fifth, conditional live spike — dialog auto-dismiss defeat, required only of an adapter that overrides its CLI's unattended dialog timeout (claude-code: CLAUDE_AFK_TIMEOUT_MS through the per-run settings env block, compat §11; codex has no equivalent knob today). "Four required live spikes" in the title and body is the original count; docs/agents/provider-authoring.md carries the authoritative checklist. The four original spikes and both tiers are otherwise unchanged.

Considered options

  • Per-adapter hand-written assertions, one test file per provider. Rejected: this is the status quo the ADR fixes — obligations pinned in claude-code's own tests drift out of sync with what a new adapter is actually held to, and a new adapter can pass review while silently skipping one. One shared entry point run against every registered adapter cannot drift adapter-to-adapter by construction.
  • Run the live spikes in CI behind a build tag. Rejected: there is no real CLI binary and no real credentials available in CI, and ADR-0008 already drew this exact line for internal/compat's live-verified pins. A build tag does not manufacture a CLI that isn't there; it only hides the gap behind a flag nobody sets.
  • A bespoke dialect validator instead of the product's own compilation paths. Rejected: a hand-rolled BRE/RE2 checker is a second implementation of what provider.CompileScrubPatterns and grep -i already do, and a second implementation can silently drift from what production actually compiles and runs — exactly the failure mode the dual-engine check exists to catch in the product path, not in a parallel one.
  • Fold the checklist into README or CONTEXT.md instead of a dedicated agents doc. Rejected: this is an agent-facing operating procedure — an ordered author path an implementing agent follows step by step — and agent-facing operating docs live in docs/agents/ (see docs/agents/domain.md), not in the project README or the domain glossary.

Consequences

  • internal/provider/providertest/conformance.go gains the Conformance(t, p, fx) entry point and the Fixture type; internal/provider/conformance_test.go gains TestConformance and becomes the one registration table every adapter — claude-code included — is entered into.
  • internal/compat/<providerID>/compat.md becomes the committed convention for a new adapter's Tier 2 record; internal/compat/compat.md is unchanged and stays claude-code's worked example.
  • docs/agents/provider-authoring.md is the new agent-facing checklist walking the author path end to end; CONTEXT.md gains a short glossary entry naming the conformance suite alongside the existing provider/seam vocabulary.
  • internal/provider/neutrality_test.go and web/src/providerNeutral.test.ts are unchanged by this ADR but are the mechanism the "core stays adapter-agnostic" hard requirement leans on.
  • Registering #2 (Codex) means: implement the seam, pass Tier 1 through the one registration table, run and record the four Tier 2 spikes at internal/compat/codex/compat.md — no change to the suite itself.