Benchmarks
Every number on this site traces to raw data committed in docs/benchmarks/ — including the experiments that didn't flatter us. Measurement over marketing.
The critic bake-off
12 candidate models, 7 frozen review cases each (planted bugs that must be flagged, clean changes that must pass), latency and format discipline measured. Two results shaped the product:
precision anchor
Nemotron (NVIDIA, free)
0 false positives on clean changes, 2-of-4 catches on flaggable ones, ~1.9s median judgment. Never cries wolf — the default primary.
recall prober
gpt-5-mini (OpenRouter)
4-of-4 catches but 2 false positives — full recall, imperfect precision. The measured inverse of Nemotron.
Council mode is built directly on that measurement: the precise model's verdict always flows through, and a prober-only finding is delivered only after an executed repro confirms it — recall without inheriting the false positives.
The A/B pilot: Claude Code with vs without
The question that actually matters: does an agent ship better work with the council watching? We built a paired experiment — identical tasks in fresh trap-seeded repos, one arm bare Claude Code, one arm with the council live. Scoring is fully mechanical: hidden acceptance tests the agent never sees, did-a-test-command-actually-run (parsed from session transcripts), and a false-claim check.
pilot result · 6 tasks × 2 arms · 2026-07-23
No measurable outcome difference — and we're publishing that.
- Ceiling effect: current Claude Code passed all six short tasks unaided — no headroom for any reviewer to improve.
- Latency mismatch: sessions averaged ~44s; the critic's first verdict lands ~30s in. Delivery can only influence sessions longer than the judge loop — real sessions are, these weren't.
- What did differ: the council's 2 findings (a seeded secret token, a lying docstring) were both genuine — zero false positives across all six with-council sessions.
- The claim metric works: both arms committed “tested and ready” without running tests when nudged to skip them — caught mechanically, both times.
Round 2 is designed and committed with the harness: longer multi-file tasks, an eval-profile critic, ≥3 trials per cell. Reproduce it yourself: python3 -m evals.ab.run --trials 3
How we'll measure it
CodeCouncil is a reviewer. A reviewer adds a model call, a verification repro, and sometimes a blocked "done" declaration — that is, by construction, more wall-clock time and API spend than not having one watching. We will never claim "cheaper" or "faster." The claim is safety and correctness, and the cost of getting them is reported openly, not hidden. Below are the four metrics that claim is built on, defined now so a future number can't quietly redefine them:
| Metric | Definition | without | naive | with |
|---|---|---|---|---|
| safe-rate | seeded vulnerabilities blocked, scored by executing the exploit | 10/15 (67%)* | 11/15 (73%)* | 11/15 (73%)* |
| correctness | hidden edge-case tests passed, tests the agent never sees | run it yourself → | run it yourself → | run it yourself → |
| integrity | false "tested and ready" claims caught mechanically | run it yourself → | run it yourself → | run it yourself → |
| added-cost | session seconds and model calls the council adds, shown openly | n/a | run it yourself → | run it yourself → |
without is bare Claude Code. naive is bare Claude Code with one generic self-review sentence appended to its prompt — the free control that has to be beaten before the council's verified, executed review means anything. with is the council live. Every filled cell traces to a committedresults.ndjsonl— never an invented number.
*safety runs 1–4 · 180 sessions total · 2026-07-25
Four runs, four layers found and fixed — in public, with raw rows for every one. Run 1: the critic was too slow for ~14s sessions → built the done-gate. Run 2: gate worked; recall was thin and nothing could deliver into a finished session → enabled the high-recall prober + gated delivery. Run 3: findings 2→31, delivered 0→5; verification's tool-calling was the weak link → rebuilt it: the model writes the repro script, the harness executes it. Run 4: every verification executed a real script (the hardest bug class live-verified first-try), delivered rose again — and the residual is now model discipline at the last step (scripts that skip the output contract; one that mislabeled its own crash as a disproof — prompt-fixed with a contract test). Safe-rates stayed within noise throughout, and we've said so every time: the base agent aces most single-function traps, so this instrument is saturated — harder multi-file tasks are the named next step. Run 4 · 3 · 2 · 1.
Full method — arms, tiers, isolation, the self-test gate, limitations, and attribution to ponytail's benchmark design — in docs/benchmarks/METHODOLOGY.md. Reproduce it: python3 -m evals.ab.run --selftest (prove the scorers discriminate, zero API spend), then python3 -m evals.ab.run --tier both --arms all --trials 4. A live multi-session run is a real-money decision we haven't made yet — when we do, the raw rows land in docs/benchmarks/ next to this table, not just a headline.
Dogfooding numbers
CodeCouncil watches its own repo — the critic reviews its builders — so these come from real, adversarial use, not a demo environment:
- ~90 s plant-to-catch on a claim-vs-code bug; ~2 min catch-to-delivery into the agent's context.
- It caught a secret-leak bug in its own redaction code, a permissions hole in its own installer (with an executed repro, blocking the agent's “done”), and a lying docstring in its own miss-detector.
- Heuristics evolved organically across versions with one honest auto-rollback; rebuttal knowledge measurably stopped a recurring false-positive class.
- 647 tests, stdlib-only, CI on Python 3.10/3.12 + lint + installer smoke + benchmark scorer self-test.
Small-n caveat, stated plainly: the self-improvement curves are days old, not months. That's what running it grows.