Why the harness matters more than the model

The same model weights score about 30% on ARC-AGI-3 as a bare model and 95.5% inside a good harness, and that gap is the number agent builders keep quoting. This page collects who says the harness matters more than the model, what each of them measured, and what the claim does not mean.

Y Combinator's Paper Club session of September 7, 2026, with the authors of Prime Agent, OpenJarvis, and QM. One hour; the history of harnesses runs from 7:00 to 17:00.

The claim

A model answers; an agent acts. The harness is the runtime that turns one into the other (definition): the tool list, the approval rules, what the model sees each turn, what survives a crash, and, since 2026, what the agent is allowed to change about itself. "The harness matters more than the model" is shorthand for a measurable fact: hold the weights fixed, change only the harness, and benchmark scores move more than most model upgrades move them.

The measurements

• ARC-AGI-3, same weights, 30% to 95.5%. A bare frontier model was verified at about 30% on the private ARC-AGI-3 set. Prime Agent, a self-improving harness from Prime Intellect, reached 95.5% with Opus 5, above the reported human-expert baseline. The YC talk walks through the runs at 30:00, and the host adds that an NVIDIA harness has since reported 100%.
• SWE-bench Pro, harness-only swap, 23% to 52%. @joelniklaus held the model fixed and changed the harness: GLM-5.2 went from 23% to 52% pass@1, and Gemma 4 26B from 15% to 36%. Latent Space's summary: swapping the harness "changed pass@1 more than many model upgrades do" (AINews, Aug 8 2026).
• Cursor's benchmark, 46% to 80%. Cursor's benchmarking research found the same model passing 46% of tasks in one harness and 80% in another.
• Rankings do not transfer. Harness-Bench measured a rank correlation of about -0.05 between harness rankings on different models. The best harness for one model says almost nothing about the best harness for another.
• Harness search beats hand design. Meta-Harness, from Stanford's IRIS lab, searched over harness code end to end and beat a state-of-the-art context manager by 7.7 points while using four times fewer context tokens.
• Two harnesses, identical weights, 18 points apart. The YC host's opening slide (1:00) shows an 18% gap between two harnesses on the same weights, the number he uses to answer the claim that harness work is not research.

The chart that opens the decision guide plots the coding-benchmark spreads.

Who says so

Measurement rows are the evidence; commentary rows show how far the claim has spread. Every link was checked on 2026-09-14. Missing an authority? Open an issue with the primary source.

What the claim does not mean

The YC talk adds the nuance the slogan drops. Timestamps point into the video.

1. Two eras, and the gains are in the second one. The static harness (7:00 to 13:00) ran from GPT-2's sampling loop in 2019 through few-shot prompts, chain of thought, tools, editable memory, skills, reflection, sub-agents, and recursive language models: a fixed program around a fixed model. The self-improving harness (14:00 to 17:00) lets the agent change its own prompt (DSPy, GEPA), its own code (Darwin Gödel Machine), or its whole harness state (Continual Harness, Meta-Harness). The recent jumps come from the second era.
2. More harness is not the claim. The right harness is. YC's QM runs on three core tools and calls itself an "AGI-anticipating harness" (57:00). Prime Agent's rule (25:00) is to expose what the model cannot do for itself, such as compaction, a persistent REPL, and programmatic sub-agents, and to drop imposed procedure like fixed plan-act-critique loops, which models now run on their own.
3. The harness is a cost lever, not only a score lever. One popular harness spent about $5,000 on ARC-AGI-3 without progress before the run was cut off (33:00), while Prime Agent finished the set. Working on context programmatically, instead of stuffing it into the prompt, is what saved the money.
4. Long-horizon failure is a harness problem. Agents "give up way too early" (57:30), so QM sets wall-clock and token budgets on goals and will not let the agent quit before the budget is spent. The QM team notes that OpenAI and Anthropic used a similar technique on open math problems.
5. Automatic self-improvement still needs a person in the loop. QM's team tried hill-climbing on their trace set and hit "main character syndrome" (53:00): each agent fixes the piece of the elephant it can see.
6. It is a pairing, not a winner. Because rankings barely transfer across models, "best harness" only means something for a given model, and the choice must be re-asked whenever the model changes.

What to do about it

• Pick the harness with the same care as the model, and re-pick when the model changes. The six questions turn this list into that decision; the two-week test drive checks it on your own repos.
• Treat the harness as a first-class variable in every benchmark you read or run. A score without the harness named is half a number.
• Budget long-horizon runs explicitly: wall-clock time and tokens, per goal.
• Prefer thin harnesses that expose capabilities over thick ones that impose procedure, and expect the right amount of harness to shrink as models improve.
• Let your agent choose. The MCP server in this repo exposes recommend and pick_harness, so a coding agent can pick a harness matched to its model and task instead of inheriting whichever one someone else benchmarked.

Part of best-of-Agent-Harnesses. Corrections and missing sources: open an issue with the primary link.

Agents can query this page's data directly: claude mcp add agent-harnesses -- uvx agent-harnesses-mcp (MCP server).