Research question and design
The experiment asks a practical systems question: when the agent harness and benchmark tasks are held fixed, how much do coding-agent quality, cost, and failure behavior change across model/provider backends?
Controlled harness
- Agent: Claude Code, held fixed as the agent harness.
- Benchmark: Terminal-Bench 2.0 task suite.
- Orchestration: Harbor.
- Full-suite design: 20 tasks × 3 attempts = 60 trials per arm.
- Workflow: inexpensive canary/smoke validation before full sweeps, followed by aggregation and evidence review.
Reviewed population
The core scope contains 15 full-suite arms (900 trials). The extended scope contains those arms plus the Phase-3-compatible Kimi K3 addendum, for 16 arms (960 trials). The extended scope is the primary public comparison here; the distinction is retained so the original core population is not silently redefined.
Quality and provider-aware cost
The primary cost shown here is the current selected provider-aware cost from the reviewed V4 authority, not the older adjusted-cost layer. Every arm has the same 60-trial full-suite denominator, so selected total cost and selected cost per attempt preserve the same ordering.
What “exact,” “estimate,” and “lower bound” mean
Exact does not always mean invoice
For some arms, exact selected cost is a provider-billed arm total. For others, it is an exact reconstruction of retained usage under reviewed provider rates. The cost basis and provider billing status fields preserve that distinction.
Aggregate evidence is not trial allocation
When only an exact provider arm aggregate is available, the report does not redistribute that total into trial or outcome buckets. Missing evidence is never converted to zero, and a broader provider-context total is not treated as exact selected-run spend.
Reviewed comparison table
The table is generated from the same public-safe artifact used by the interactive frontier. Cost/clean-success is total selected cost divided by reviewed clean successes; it is an efficiency ratio, not an allocation of provider spend to successful trials.
| Model / arm | Provider | Scope | Successes | Pass rate | Selected cost | Relation | Cost / attempt | Cost / clean success | Cost basis | Allocation |
|---|
Failure modes and trajectory findings
Aggregate pass rate hides materially different operating behavior. The retained Phase 3 behavior profile showed that exception incidence varied substantially across arms: Gemini Flash was particularly exception-heavy, while Haiku’s failures were predominantly normal benchmark failures rather than exceptions. Sonnet, GLM, Qwen, Grok, Kimi K2.6, and Gemini Pro also had notable exception counts in the reviewed core runs.
Visible-token patterns were heterogeneous as well. The behavior profile grouped GLM, Grok, Qwen, and Kimi K2.6 among lower-token success patterns, while several Anthropic and DeepSeek arms showed higher-token success patterns. Two suspect no-op cases were flagged in each of the historical GPT-5.5 and Gemini Flash reviews. These are diagnostic signals, not proof of a single causal mechanism.
The Kimi K3 addendum illustrates why trajectory review matters: it recorded 47 raw successes but only 44 clean successes, because three timeout trials still carried success signals. Ten trials hit agent timeouts. Treating “exception” and “failure” as synonyms would lose that distinction.
Behavioral differences observed
- Quality leaders were not necessarily cost leaders. The highest pass-rate extended arm was not the least expensive selected run.
- Provider/model behavior differed in how work failed. Some arms concentrated failures in exceptions/timeouts; others produced more ordinary benchmark failures.
- Token consumption was not a simple proxy for quality. Higher- and lower-token success patterns both appeared among mid-pass-rate arms.
- Accounting artifacts themselves differed by provider. Some providers supported exact arm-level billing joins, while others required rate reconstruction, cross-checks, or lower-bound qualification.
Evidence and provenance architecture
The benchmark separates historical harness-recorded/adjusted cost from the current provider-aware selected-cost layer. The current comparison reconciles retained trial/trajectory usage, provider-rate evidence, provider billing exports or dashboards where available, and run-allocation qualifications. That separation prevents a newer provider total from silently overwriting frozen benchmark outcomes or being allocated more precisely than the evidence supports.
The website is deliberately a consumer of the reviewed benchmark authority. A deterministic generator validates the pinned benchmark commit, authority path, authority Git blob, schema, reviewed date, and scope cardinality before emitting the stripped public JSON and SVG preview.
Limitations
- This is a 20-task Terminal-Bench suite with three attempts per task, not a universal ranking of coding models.
- Backend/model performance is conditional on Claude Code as the fixed agent harness and on the benchmark/runtime configuration used for these runs.
- Provider evidence quality is heterogeneous. Exact, estimate, and lower-bound relations must remain visible in cost comparisons.
- Some provider totals are aggregate evidence without trial-level or outcome-level allocation; the report intentionally avoids manufacturing those allocations.
- Historical behavior/trajectory diagnostics were reviewed on the retained runs and are descriptive, not a causal decomposition of model capability.
- The Kimi K3 run is an extended Phase-3-compatible addendum, not a retroactive change to the 15-arm core.
Reproducibility and public artifacts
Authority: results/phase3/reporting/phase3_current_reviewed_comparison_20260825.json
Historical diagnostic sources: phase3_arm_behavior_profile_20260716.tsv and Kimi K3 addendum summary. They supplement the current V4 cost/quality authority rather than replace it.