What Is an Agent Harness? Research, Open-Source Projects and Industry Practices

Two agents can use the exact same model and score wildly differently on the same benchmark. OpenAI measured this directly: on SWE-bench Lite, GPT-4 scored 2.7% with one harness and 28.3% with another — a 10x gap created entirely by the harness. As the industry shifted from "which model is smartest" to "which agent actually completes the job," the agent harness became the layer where reliability is won or lost. Keep reading to learn what an agent harness is, what the newest papers and benchmarks say about harness design, which open-source harnesses are worth your time, and how the major AI labs engineer their own — including Kimi K3.1 Pro, the open-weight model Moonshot AI pairs with its own harnesses.

An agent harness is the surrounding infrastructure that lets a large language model operate as an agent inside a real environment. Where the model produces tokens, the harness does everything else: it builds the prompt at every step, defines which tools exist and how they are called, executes actions in a sandbox or terminal, feeds observations back to the model, enforces step and cost limits, and logs the full trajectory for evaluation or debugging.

The term comes from the evaluation world — a "test harness" is the rig that runs a system under controlled conditions. In agentic AI, the same idea was extended to production: a harness is both the rig used to measure an agent (as in SWE-bench or Terminal-Bench) and the harness used to run an agent (as in SWE-agent, OpenHands or Claude Code).

Anthropic's own SWE-bench research puts it plainly: the benchmark "doesn't just evaluate the AI model in isolation, but rather an entire 'agent' system," where the agent is "the combination of an AI model and the software scaffolding around it." Their upgraded Claude 3.5 Sonnet reached 49% on SWE-bench Verified not with a complex framework, but with a minimal harness of one prompt and two general-purpose tools (a Bash tool and an Edit tool).

For years, leaderboard gains were attributed to model scale. Recent research broke that assumption from two directions.

First, the "less is more" result: in July 2025, the SWE-bench team released mini-SWE-agent, a harness of roughly 100 lines of Python with no tools other than bash, a completely linear message history, and every action executed as an independent subprocess.run call. It reached benchmark scores competitive with far more complex frameworks (covered in the research section below). The lesson was provocative: once the model is strong enough, an extremely simple, predictable harness can beat a heavyweight one. It is already used as a baseline by Meta, NVIDIA, IBM, Princeton and Stanford.

Second, the "more is less" result: a poor harness — noisy tool outputs, unbounded loops, bad error recovery — can make a frontier model look mediocre. OpenAI found that cleaning up SWE-bench's flawed tasks (detailed in the OpenAI section below) more than doubled measured model performance.

This is why harness engineering has become a discipline of its own. The harness determines:

• What the model sees: prompt structure, tool schemas, how much of the environment state and history is included in the context window.

• What the model can do: which tools are available (shell, file editor, browser, search), and how actions are parsed and validated before execution.

• How failures are handled: retries, timeouts, context compaction when the window fills up, and graceful termination.

• How work is measured: trajectory logging, grading, and cost accounting — without which there is no evaluation and no improvement loop.

Whether it is a research benchmark rig or a production coding assistant, nearly every agent harness implements the same core loop:

1. Task intake: read the issue, instruction or goal, and spin up an isolated environment (usually a Docker container or VM snapshot).

2. Prompt assembly: combine the system prompt, tool definitions, task description and the running history of actions and observations.

3. Action parsing: translate the model's output into a concrete tool call — a bash command, a file edit, an API request — and reject malformed actions.

4. Execution and observation: run the action in the environment, capture stdout/stderr or UI state, and truncate it to fit the context budget.

5. Control flow: enforce step limits, cost limits and stopping conditions; decide when to ask a human or end the episode.

6. Recording and grading: save the full trajectory and evaluate the final state — for example, by running the repository's test suite in SWE-bench.

Design choices at each step compound. Truncation strategy alone can swing scores noticeably on long-horizon tasks, which is why serious teams treat the harness as versioned, testable software rather than glue code.

The academic literature on agent harnesses has converged on a small number of hard-won conclusions. Rather than catalog every benchmark, this section traces the research questions that forced the field to take the harness seriously.

Before the papers, a quick primer. A benchmark is a fixed set of tasks plus a scoring rule, designed so different systems can be compared on identical ground. In agent evaluation, a benchmark has three moving parts: a task set (real GitHub issues, terminal commands, web tasks), an execution environment (a Docker container, a browser, a terminal), and a grader (a test suite, a state check, a human review). The harness sits between the model and all three — it is the variable that decides whether the model's ability shows up in the score.

How scores are actually measured matters more than the headline number. In SWE-bench, an agent produces a code patch; the benchmark then runs two test suites: FAIL_TO_PASS (tests that were broken before the fix and must now pass) and PASS_TO_PASS (tests that worked before and must not break). Only if both pass is the task counted as resolved. This is why harness features like test execution and diff generation are not optional extras — they are where points are won or lost.

Three rules for reading any benchmark score:

1. Ask what the score hides. A model trained on public GitHub data may have seen the very issues it is being tested on — data contamination can inflate scores without any real capability gain. Reproducible environments and held-out task sets are the antidote.

2. Beware Goodhart's law: when a measure becomes a target, it ceases to be a good measure. A harness tuned to ace one benchmark may be learning benchmark-specific tricks rather than general capability — which is why serious teams evaluate on more than one.

3. Check the harness, not just the model. As the rest of this section shows, the same model can score 2.7% or 28.3% depending on the scaffolding. A score without a named harness is an anecdote, not a measurement.

The original SWE-bench paper ("Can Language Models Resolve Real-World GitHub Issues?") introduced the now-canonical evaluation: give a model a real GitHub issue and the full repository, and grade whether its patch makes failing tests pass. Its most cited result is often misread: the best model at the time, Claude 2, solved only 1.96% of tasks. The authors' own framing was that the benchmark tested "complex reasoning that goes far beyond traditional code generation" — but the follow-up work made clear that a large part of the gap was not reasoning ability, it was the absence of a harness that could navigate a repository, run tests and iterate. In other words, SWE-bench did not just measure models; it revealed that the harness around them was the missing piece.

The SWE-agent paper ("Agent-Computer Interfaces Enable Automated Software Engineering") is the field's founding document for harness design. Its core claim is that interface design is a first-order variable: the authors introduced the term Agent-Computer Interface (ACI) and showed that a purpose-built interface — a linter-guarded file editor, concise shell feedback, guardrails on malformed commands — raised pass@1 on SWE-bench from low single digits to 12.5%, at the time state of the art. The paper's lasting contribution is the argument that "LM agents represent a new category of end users" who need software built for them, just as humans need IDEs. Every subsequent harness paper cites this result.

Two 2024 papers extended harness research from code repositories to the full computer. OSWorld built the first scalable real-computer environment across Ubuntu, Windows and macOS, with 369 tasks spanning web and desktop apps. Its headline number — humans complete 72.36% of tasks, the best model 12.24% — identified GUI grounding (translating model output into correct clicks and keystrokes) as the dominant failure mode, a harness problem, not a knowledge problem. WebArena took a different angle: a self-hosted, fully reproducible web environment (e-commerce, forums, GitLab, CMS) where GPT-4 reached only 14.41% end-to-end success versus 78.24% for humans. Both papers established that harness evaluation must be execution-based (checking the final state of the environment), not pattern-matched on model outputs.

Sierra's τ-bench ("A Benchmark for Tool-Agent-User Interaction in Real-World Domains") made a subtler point: in production, the harness does not just wrap the model and the environment — it also mediates the user. The benchmark simulates dynamic conversations between a language-model user and an agent with domain APIs and policy guidelines, then grades the final database state. Its key metric, pass^k (success across k independent trials), exposed how inconsistent agents are: GPT-4o solved fewer than 50% of tasks once, and pass^8 dropped below 25% in retail. The lesson for harness builders is that reliability is a distributional property, not a single-run property — a harness that works once is not a harness that works.

METR's "Measuring AI Ability to Complete Long Software Tasks" introduced the 50%-task-completion time horizon: the length of task (measured by human completion time) that a model finishes with 50% reliability. Frontier models went from minutes to roughly an hour between 2019 and 2025, doubling about every seven months. For harness engineering this is the most consequential finding of the decade: the harness that constrains a 2024 model is dead weight for a 2026 model. Anthropic's framing is that every component in a harness encodes an assumption about what the model can't do — and those assumptions go stale as models improve.

The original benchmark has grown into an ecosystem that operationalizes these lessons:

• SWE-bench Verified (with OpenAI, 2024): a 500-instance human-filtered subset that removed underspecified or unfairly graded tasks — 68.3% of the original pool was filtered out, which more than doubled measured scores and became the default leaderboard.

• SWE-smith (2025): a pipeline for generating synthetic training tasks so models can be trained for agentic harnesses, not just evaluated on them.

• mini-SWE-agent (2025): the ~100-line harness that now exceeds 74% on SWE-bench Verified, demonstrating that a minimal, bash-only loop with a linear history is enough once the model is strong — and becoming the community's default baseline.

• Terminal-Bench (Stanford × Laude, 2025): terminal-task evaluation built on the Harbor framework, pushing harnesses beyond code editing into system administration.

• CodeClash (2025) and ProgramBench (2026): goal-oriented and from-scratch software creation, extending harnesses from single-issue fixes to open-ended development.

The through-line across all of this work is consistent: the harness is not plumbing, it is the object of study. Report the harness with the score, keep the environment reproducible, and suspect any result whose harness cannot be inspected.

Open-source harnesses are where most teams start. The projects below combine large GitHub communities (stars in the tens of thousands for the leaders) with active maintenance:

When choosing among them, the community heuristic is: use the minimal harness that expresses your task (mini-SWE-agent philosophy), and add complexity only when a measured failure demands it. DeepSeek Harness occupies the opposite end of the spectrum — a fully plugin-driven platform for teams who want every layer replaceable — but the two share a common discipline: a durable, append-only session log as the single source of truth for what the model saw.

The frontier labs treat harness engineering as core IP, and several have published their thinking in unusual detail.

Anthropic has published the most complete public record of harness engineering, and its architecture — detailed across Building Effective Agents (2024), the SWE-bench harness post (2024), and Managed Agents (2026) — can be read as three concentric layers.

The execution layer is deliberately minimal: a Bash tool for running commands and an Edit tool for viewing and modifying files. The design effort went into making those tools error-proof — requiring absolute paths, truncating long outputs, and applying edits only on a unique string match — so the model rarely has to recover from avoidable tool failures.

The session layer is a durable, append-only event log that lives outside the model's context window. Every action and observation is recorded there; the model's context is a projection of that log, not the log itself. This separation is what allows a failed harness process to be rebooted and resumed mid-task, and what makes the harness disposable "cattle" rather than a pet.

The orchestration layer — what Anthropic calls the "meta-harness" — decouples the "brain" (the model and its loop) from the "hands" (sandboxes and tools) via explicit interfaces. Because the brain no longer lives inside the execution container, sandboxes can be provisioned on demand and credentials can be held in a vault that the sandbox never touches. The payoff they report is structural, not cosmetic: p50 time-to-first-token dropped ~60% and p95 over 90%, because sessions no longer wait for containers they may never use.

Three documents trace this evolution: Building Effective Agents (2024), the SWE-bench harness post (2024), and Managed Agents (2026).

OpenAI partnered with the SWE-bench team to create the human-validated Verified split precisely because harnesses need clean tasks: their annotation campaign (93 developers, 1,699 samples) found that 68.3% of the original benchmark's tasks were problematic, and removing them more than doubled measured performance. The same post notes that harness choice alone moved GPT-4 from 2.7% to 28.3% on SWE-bench Lite — a spread they cite to argue that evaluations must account for progress in community-built harnesses.

On the open-source side, the Codex CLI ships as a complete reference harness. Its architecture is layered in a way that mirrors the other systems in this section: a core agent loop handles planning and tool calls; a sandbox layer isolates execution with configurable policy per task; a session layer persists conversations so they can be resumed, forked or compacted; and a tool layer exposes both local shell access and MCP-based external tools. Rather than hard-coding a single interface, the same engine is exposed through three surfaces — a terminal UI for interactive work, a JSON-RPC app server for rich clients like VS Code, and a remote code-execution service for cloud sandboxes — so the harness can run locally or be driven programmatically. The design makes the same trade-offs the research literature recommends: minimal loop, strong sandbox, explicit context management, and a clear boundary between the model and the outside world.

With the Agent Development Kit (ADK) and the Agent2Agent (A2A) protocol — launched in April 2025 with more than 50 technology partners including Atlassian, Salesforce, SAP and ServiceNow — Google is productizing the harness layer. A2A standardizes how agents advertise capabilities (via JSON "Agent Cards"), manage long-running tasks, exchange artifacts and negotiate UI formats, so that harnesses become interoperable infrastructure rather than bespoke glue. It is explicitly designed to complement Anthropic's Model Context Protocol (MCP), which standardizes the tool side of the harness.

Microsoft's AutoGen and Magentic-One ship harnesses where an orchestrator agent manages a ledger of facts and progress — a published design for preventing the unbounded loops that plague naive harnesses. Meta and Hugging Face's GAIA created a general-assistant benchmark whose harness grades real-world questions requiring web navigation and file handling; Hugging Face's Open Agent Leaderboard continues the push for comparable, harness-aware rankings.

DeepSeek Harness (dsh) takes the meta-harness idea and makes it composable. Its architecture is built on the Cordis framework, and its central claim is that every part of the harness is a replaceable plugin — including the model adapter, the tool registry, the session log and the agent loop itself.

The design rests on three primitives. First, a durable session log that is the single source of truth for what the model saw; anything that reaches the model must be reconstructable from the log, enforced as a runtime invariant. Second, capability seams: a swappable provider interface between the agent and the outside world, so that pointing the filesystem and subprocess providers at a remote sandbox moves Bash, terminal and language-server access with them, without forking the harness. Third, an explicit turn/step lifecycle with named extension events (agent/pre-step, tools/execute, agent/turn-stopping) that let plugins intercept or rewrite the loop without modifying core code.

The result is the closest open-source realization of Anthropic's meta-harness: a harness where the structure is not a fixed framework but a composition of plugins that can be swapped from configuration. With over 170k GitHub stars in its first months, it signals that the agent harness is becoming a product category in its own right.

The shared pattern across all of them: keep the loop simple, sandbox everything, log every step, and never trust a score without the harness that produced it.

Everything in this section so far — the minimal loops, the durable session logs, the sandboxed execution, the plugin architectures — shares one assumption: the agent's environment is a computer. A terminal, a browser, a filesystem, a repository. The harness exists so the agent can operate that machine reliably.

But the research trajectory traced above points somewhere else. τ-bench put the user inside the harness. METR showed agents taking on work that spans days, not minutes. OSWorld and WebArena moved the environment from a clean repo to the messy real world. The direction is consistent: as agents move from coding tools to actual work assistants, the environment they must navigate is no longer just silicon — it is people. Users with expectations, teams with history, organizations with permissions and institutional knowledge.

That shift lands on one harness subsystem above all others: context management. A terminal session has one operator and a bounded history; a long engagement has ten stakeholders, months of decisions and a repository that keeps growing. A filesystem is passive; a working relationship is a stream of implicit commitments, half-finished thoughts and choices made in passing. All of it has to be summarized, compacted or dropped before it reaches the model, and every one of those decisions is a place where the agent loses something the task needed.

A harness can compress, but it cannot compress arbitrarily. Which is why the size of the context window stopped being a model specification and became a harness design parameter.

Moonshot AI's harness portfolio maps the anatomy defined earlier in this article onto one managed stack. For the Agent × Computer problem, its harnesses extend the standard loop: Kimi Browser Extension drives a real browser session via the Chrome DevTools Protocol instead of a sterile sandbox, an Ask-before-acting seam enforces consent at the filesystem level, and an Agent Swarm scales orchestration to 300 sub-agents for long-horizon tasks.

What drives that loop is Kimi K3.1 Pro — the first model in the K3.1 series and Moonshot AI's most capable 3T-class open-weight model to date, with a 1,048,576-token context window. Three of its properties bear directly on harness design:

• A window large enough to postpone compaction. A million tokens holds a working repository, the running trajectory and the reference documents at the same time. The harness still truncates — it always has to — but the threshold at which summarization starts destroying information moves far enough out that long-horizon tasks stop quietly degrading halfway through.

• Reasoning effort as a harness knob. K3.1 Pro always thinks, and reasoning_effort accepts low, high or max, with max the default. That turns the model's reasoning cost into a per-task budget the harness sets, rather than a fixed price of admission.

• Tools that arrive mid-episode. Dynamic tool loading lets the harness inject a complete tool definition inside a system message, so a tool becomes available from that message onward — no restart, and no paying for its schema on every turn of a conversation that never uses it.

Moonshot reports the harness alongside the model, which is the discipline the rest of this article argues for. On Terminal-Bench 4.0 — which scores coding agents on realistic, resource-constrained terminal tasks under strict task-specific tests — K3.1 Pro reaches 65.7%, ahead of GPT-6 Astra (58.2%) and just behind Claude Opus 5.5 (66.4%). On Kimi Code Bench 3.0 Long-horizon, an internal benchmark for multi-step execution, tool use and end-to-end problem solving, it scores 61.2% against K3's 55.95%. The documentation is just as explicit about the coupling running the other way: sampling parameters such as temperature and top_p are fixed and must be omitted from requests, and the complete assistant message has to be returned unchanged across turns — a harness that rewrites the message drops the reasoning the next tool call depends on.

The same model runs the whole surface. You can try the managed web version through Kimi Agent, run the full local harness with Kimi Work, work in the terminal with Kimi Code, or call kimi-k3.1-pro directly through the Kimi API. A kimi-k3.1-pro-highspeed variant trades roughly double the token price for two to three times the output speed.

Whether you are evaluating models or shipping a product, the same checklist applies:

1. Start from the task, not the framework. If your agents edit code, a bash-plus-editor harness like SWE-agent's ACI is proven. If they drive GUIs or the live web, you need real-session observations and click-level actions.

2. Make environments reproducible. Docker images, pinned dependencies and seeded data are the difference between a benchmark and an anecdote.

3. Budget the context window deliberately. Observation truncation, history summarization and compaction are harness features that directly move scores and costs — and the reason long-context models reduce how much the harness has to throw away.

4. Enforce hard limits. Step caps, wall-clock timeouts and dollar budgets per episode keep agents safe and evaluations comparable.

5. Log full trajectories from day one. Every serious improvement loop — human review, failure clustering, fine-tuning — consumes trajectory data. Anthropic's Managed Agents and DeepSeek Harness both go further: keep the session log outside the harness loop as an append-only event stream, so the harness itself is disposable and the model's context is always reconstructable.

6. Evaluate on more than one harness. If your agent only works inside one harness, you have built harness-specific behavior, not capability.

7. Keep credentials out of the sandbox — or out of the cloud entirely. Treat the agent's execution environment as untrusted; hold tokens in a vault and inject them through a proxy, or run the whole session locally so credentials and page content never leave the device — the approach Kimi takes with its local harnesses.

The agent harness has quietly become the most important layer of the agentic stack. Research from SWE-bench, mini-SWE-agent, Terminal-Bench and METR shows that harness design can rival model upgrades in impact; open-source projects like DeepSeek Harness, OpenHands, SWE-agent and Aider put production-grade harnesses within everyone's reach; and Anthropic, OpenAI, Google, Microsoft and Moonshot AI all now treat harness engineering as first-class work.

But the deeper shift is conceptual. The first generation of harnesses solved Agent × Computer: giving a model reliable hands for a terminal, a browser, a filesystem. The next generation is solving the context problem those environments create — work that spans days, stakeholders and repositories no summarizer compresses losslessly. The harness is evolving from scaffolding into an operating environment, and the line between what the harness has to manage and what the model can simply hold keeps moving. Kimi K3.1 Pro's one-million-token window is the clearest current example: a model change that deletes work the harness used to have to do.

Whether you build your own, adopt an open-source framework, or run a production harness, the teams that treat the harness as engineered, versioned, measurable software are the ones whose agents actually finish the job.