best-of-Agent-Harnesses

Harness engineering: what it is, and the tools for each practice

Harness engineering means building everything around an AI model that turns it into a dependable agent: what it sees, which tools it gets, what it may do without asking, and how it checks its work. The model stays the same; you change its surroundings until a mistake it made once stops happening.

Essays explain the idea. This page maps each practice to the open-source projects in this list that do it, with the list's own ratings: autonomy (how much it does without asking you), recovery (what survives when a run dies), and complexity (how much you take on by adopting it). A file format, library, or add-on with no agent loop of its own carries only a complexity rating.

Where the term came from

Mitchell Hashimoto named the practice in My AI Adoption Journey on February 5, 2026. In a step called Engineer the Harness, he wrote, "I've grown to calling this 'harness engineering.'" His rule: each time an agent makes a mistake, build a fix so it never makes that mistake again. The fix is a new line in AGENTS.md (a plain file of instructions the agent reads) or a small tool the agent can run. He said he was not trying to coin a term.

Six days later, on February 11, OpenAI published Harness engineering: leveraging Codex in an agent-first world by Ryan Lopopolo. Over five months, a small team built an internal product with no hand-written code: about a million lines and roughly 1,500 merged pull requests, all written by Codex. The engineers' job became designing the environment, stating intent, and building feedback loops.

LangChain's The Anatomy of an Agent Harness (March 10) gave the short formula: agent = model + harness, where the harness is all the code, settings, and logic that is not the model. Birgitta Böckeler of Thoughtworks wrote Harness engineering for coding agent users (April 2), and Lilian Weng's Harness Engineering for Self-Improvement (July 4) took it into research. The idea is older than the name: SWE-agent (May 2024) showed that how tools are presented to a model changes what it can do. The measurements behind the idea have their own page.

Context delivery

Context is what the model sees each turn, and the common mistake is one giant instruction file. OpenAI's team found it crowded out the task and went stale fast. Their fix was a short AGENTS.md, about 100 lines, that works as a table of contents for deeper docs kept in the repo. Load details only when a task needs them (Context files has more), and shrink bulky tool output first.

Check yours: agents-md-checker shows which instruction files each coding agent loads.

Tool design

A tool's shape matters as much as how many tools there are. Anthropic's guide to writing tools for agents (September 11, 2025) advises a few well-chosen tools instead of one for every call a service offers, plus clear names, short responses, and error messages that say what to do next. Claude Code caps a tool response at 25,000 tokens (the units models read and bill by) unless told otherwise. The model reads every tool's description on each request, so some projects load tools only on demand.

Check yours: tool-design-checker grades your MCP tools the way a model reads them.

Permissions and approvals

Permissions set what the agent may do without asking you. Asking before every step sounds safe, but people stop reading what they approve; Anthropic calls it approval fatigue. Its sandboxing write-up (October 20, 2025) moves safety into fixed limits instead: only certain folders, only approved servers. Anthropic reports this cut permission prompts by 84% in internal use. Both limits matter: with folder limits alone, a hijacked agent can still send your keys out over the network.

Check yours: sandbox-check shows what your agent can reach, and guardrail-tester tries about 90 dangerous commands against your rules.

Verification loops

An agent that cannot check its own work hands the checking to you. Hashimoto found that an agent with a way to verify its work usually fixes its own mistakes. OpenAI's team had Codex review its own pull request, ask other agents to review it, and repeat until every reviewer was satisfied. Böckeler sorts these controls into guides, which steer the agent before it acts, and sensors, which check after. Start with checks that give the same answer every time, such as tests and linters, and add a second model as reviewer only where no fixed check exists.

Check yours: claim-check finds the times your agent said tests passed without a passing run.

Memory and state

A new session starts with no memory of the last one, so anything that matters must live outside the model. Anthropic's write-up on long-running agents (November 26, 2025) keeps that state in plain files and git: a progress log, a feature list where every feature starts out marked as failing, and a commit at the end of each session. The next session reads them before it changes anything. The memory layers comparison goes deeper.

Recovery after a crash

Runs die: rate limits, crashed sandboxes, closed laptops. Recovery decides whether the agent starts over, retries the step, resumes the session, or continues from saved state even after a restart, which this list calls durable. Those Anthropic agents use git to undo a bad change and run a quick end-to-end test before new work, so a half-finished session gets fixed first. This is where the list is thinnest: of the 52 projects built to run with nobody watching, 22 can only start over or retry a step, and just 5 are durable.

Observability

Observability means seeing inside a run, for you and for the agent. OpenAI gave each copy of the app its own short-lived logs, metrics, and traces that Codex could search, so a goal such as startup under 800 milliseconds became one the agent could check itself. For people, tracing tools record every step and tool call, so you can find where a run went wrong and what it cost.

Check yours: session-waste-report shows which agent habits cost the most, and runaway-guard stops a session that loops, keeps failing, or passes a dollar limit.

Measuring the harness

Every change to the harness is a guess until you measure it. LangChain's Better Harness recipe (April 8, 2026) changes one thing at a time and keeps a change only if new test tasks pass and no old one breaks, then confirms the gain on tasks held back from tuning. Measure with the model you will use: Harness-Bench (May 2026) found that results depend on the pairing of model and harness.

Check yours: harness-test-drive runs Claude Code, Codex, and Gemini CLI on tasks from your own git history (full trial).

Harness engineering vs picking a framework

Picking a framework is a one-time choice of runtime: LangGraph or crewAI, Codex or Cline. Harness engineering is the work that continues after the pick: every rule, check, tool, and limit you add because the agent got something wrong. Böckeler draws the same line between the harness built into a coding agent and the outer harness its users build. The pick still matters: the runtime sets the ceiling on autonomy and recovery, and rule files cannot add recovery a runtime lacks. Most of the outer harness moves with you: AGENTS.md files, skills, MCP tools, and test tasks work across runtimes, so keep them in plain files.

Where to go next

To choose the runtime, start with How to pick a harness: six questions that cut this list to two or three candidates. To let your coding agent do the lookup, add the MCP server: claude mcp add agent-harnesses -- uvx agent-harnesses-mcp. Its pick_harness tool filters by the same ratings used on this page.

Part of best-of-Agent-Harnesses. Sources checked 2026-10-23; counts from the list's data of 2026-10-18. Corrections: open an issue with the primary link.

Agents can query this page's data directly: claude mcp add agent-harnesses -- uvx agent-harnesses-mcp (MCP server).