Why we don't trust the model to grade its own homework

Most "AI fixes your CI" demos end the same way: the model edits a file, prints a confident summary ("This should resolve the failing test"), and stops. Nobody actually ran anything.

That's the gap between a demo and a tool you'd let touch production code.

The rule: success is a subprocess exit code

code-agent is built around one non-negotiable rule: a task is only "done" when your own verify command exits 0. Not when the model says it's confident. Not when the diff looks plausible. When pytest -q, go test ./..., or whatever already gates your CI comes back green.

Plan → Act (edit files) → Observe (run verify_cmd) → Judge (exit code) → retry or done

If the LLM claims success but the subprocess disagrees, the subprocess wins. Every time.

Why this matters more than model choice

Teams evaluating AI coding agents usually ask "which model is smartest." The better question is "what happens when the model is wrong?"

• A smarter model that self-certifies success will still occasionally hallucinate a passing test.
• A weaker model wrapped in a verify gate will retry, narrow its fix, or honestly report "could not verify" — and never open a PR on a false green.

We'd rather ship an honest failure than a confident lie.

What this looks like in practice

• No draft PR without a green verify. code-agent experts run bug-fix --verify-cmd "pytest -q" --publish will not publish anything unless the exit code is 0.
• No editing your CI workflows to cheat the gate. .github/workflows/** is explicitly forbidden — a fix that "passes" by weakening your pipeline isn't a fix.
• Fail closed on ambiguity. If the verifier output is malformed or unclear, the run fails rather than guessing "probably fine."

This is why we call it a verify-gated CI agent, not an autonomous one. Autonomy without a ground-truth check is just faster hallucination.

Read more: How it works · Trust & guardrails