The Agent Harness Is the Product

Put the same coding model in two repositories and it can look like two different products.

In the first, the agent guesses how to build the project, receives twenty thousand lines of failing output, can edit every test, and has broad credentials. In the second, it finds a current map of the codebase, receives the exact task context, uses narrow tools, gets a precise failure within seconds, and cannot cross important boundaries.

The model matters. The system around it decides whether that capability becomes useful work or an expensive source of surprises.

That system is the harness.

More Than a Prompt Wrapper

A harness is the operational boundary around an agent. It determines:

• model and loop: which model acts, how turns are scheduled, and how retries work;
• context: what instructions, repository knowledge and task evidence enter each turn;
• tools: which actions the model can request and how results are represented;
• permissions: which files, services, credentials and external effects are allowed;
• state: what persists across turns, interruptions and fresh context windows;
• evaluation: which tests, checks and reviewers decide whether work may advance;
• termination and recovery: when the run succeeds, stops, escalates or rolls back.

Calling this a harness does not require a new platform. It may begin as an AGENTS.md, a few repository scripts, filesystem permissions and CI. Mitchell Hashimoto describes his own harness engineering in similarly practical terms: when an agent repeats a mistake, he improves the repository instructions or builds a tool, such as a focused test or screenshot script, so the next run can avoid or detect it.

That is a useful definition because it starts with observed failure rather than a shopping list.

A Field Report, Not a Universal Benchmark

OpenAI describes building an internal product through Codex agents. Over five months, the repository grew to roughly one million lines across product code, infrastructure, tooling and documentation. The article reports about 1,500 merged pull requests, initially driven by three engineers, with hundreds of internal users.

Those figures deserve care. They are a first-party report about one internal project, not evidence that any three engineers can reproduce the result. A million lines is not a quality measure, and the article does not establish how much of the result would transfer to an existing codebase.

The useful part is what they say the engineers worked on: environments, repository knowledge, architectural constraints, testing and feedback. Humans could review pull requests, but review was increasingly handled agent-to-agent. The claimed output depended on making the repository legible and the verification loop executable.

In other words, the result cannot be attributed to the model alone. The team built a system in which the model could repeatedly find the right context, act, receive a meaningful refusal and try again.

Context Is a Product Surface

A model cannot use a decision it never sees. It also cannot reliably find one buried among three years of obsolete plans.

The context layer should answer the immediate questions of the task:

• What outcome is required?
• Which part of the system owns it?
• Which interfaces and invariants must remain stable?
• What commands establish the baseline and verify the change?
• Which decisions have already been made?
• Where must the agent stop and ask?

This knowledge needs an owner and a lifecycle. Instructions decay when commands change. Architecture maps become fiction when imports cross their stated boundaries. Examples can teach an obsolete pattern long after the team has rejected it.

More context is not the cure. Dumping an entire wiki into every turn spends tokens, hides local rules and increases the chance that old guidance wins through repetition. Prefer progressive disclosure: a small repository map points to current, local detail; path-specific instructions appear only when that path is in scope; the task carries the decisions needed for this change.

The cost is editorial. Someone must remove stale documents, resolve contradictions and keep the useful route through the repository short. That work feels less productive than generating another feature. It often improves far more future work.

Tools Shape the Agent’s Possible Moves

A shell is powerful and ambiguous. A focused command can be narrow and informative.

Compare “check the service” with a repository command that starts the required dependencies, exercises one scenario, returns structured failures and cleans up afterwards. The second tool does more than save prompt text. It encodes how this system is meant to be operated.

Good agent tools tend to have familiar properties:

• one clear responsibility;
• discoverable help and examples;
• deterministic inputs where possible;
• concise, structured output;
• meaningful exit status and error messages;
• idempotence or an explicit warning when repetition is unsafe;
• a visible record of consequential effects.

These are also properties of good developer tools. There is no reason to maintain one obscure build path for humans and a separate collection of fragile wrappers for agents. Improve the repository’s ordinary interfaces first. Add an agent-specific tool when the model genuinely needs a safer abstraction or machine-readable result.

I would reject the opposite design, where every available integration is exposed because the model might find it useful. Tool names, schemas and results consume context. Broad tool access also expands the number of ways a mistake can escape the repository. Capability without a task is attack surface with documentation.

Permissions Are Part of Behaviour

An instruction saying “do not deploy” competes with everything else in the context. An identity that cannot deploy makes the boundary real.

Permissions decide the maximum consequence of a bad inference, a malicious dependency, prompt injection in untrusted content, or a simple command aimed at the wrong environment. Treat them as part of the product’s behaviour, not as installation detail.

Start with the smallest authority that can complete the job. Separate read, propose and apply operations. Use a dedicated identity rather than a developer’s full credentials. Put approval immediately before an irreversible or external action, where the reviewer can see the exact target and payload. Make revocation and recovery boring.

Filesystem boundaries matter too. A documentation task does not need access to production secrets. A test-fixing agent should not be able to rewrite protected acceptance tests. A repository cleanup should not discover that the user’s home directory counts as temporary state.

This restraint costs convenience. Narrow credentials must be created and rotated. Sandboxes can make legitimate work awkward. Approval points interrupt autonomous runs. The answer is not to remove every boundary; it is to place boundaries according to blast radius and improve the safe path until it is easier than the dangerous one.

State Must Survive, but Not Become a Dump

Agent turns are temporary. Work is not.

Useful state belongs in inspectable systems: Git for code and history, an issue or plan for remaining work, CI for verification results, an event log for external operations. A fresh context window should be able to recover what was attempted, what changed, which checks ran and why the work stopped.

Conversation transcripts can help with forensics, but they are not a substitute for decisions. They contain false starts, untrusted input, superseded instructions and a great deal of noise. Preserve the rationale and evidence needed to continue; do not make the next agent replay an archaeological dig through every token.

State also needs concurrency rules. Even before we introduce multiple agents, a user can send a new instruction while a run is changing files. Does the harness queue it, steer the current run, cancel and restart, or allow two loops to race? Each choice needs explicit rules for protecting shared state.

Recovery is the other half of state. Checkpoints, small commits, reversible migrations and idempotent tools let a failed run stop without turning cleanup into a second project. “The agent can try again” only helps if the first attempt did not corrupt the starting point.

Evaluation Must Be Outside the Sales Pitch

The harness needs fast backpressure for local correction and risk-tiered review for judgement. It also needs a definition of completion that the acting agent cannot quietly rewrite.

Different layers answer different questions:

compiler and types     is this structurally valid?
focused tests          does the changed behaviour match examples?
contracts              do boundaries still agree?
runtime exploration    does it work in a real process or browser?
security checks        did the change introduce a known dangerous pattern?
human review           are the intent, trade-offs and residual risk acceptable?
production signals     does reality agree with the pre-release model?

An agent’s judgement of its own prose is a weak signal, while an independent test can establish something stronger. A separate reviewer with a different rubric may find further problems, but that review is still probabilistic. Deterministic checks and human judgement remain necessary where their failure classes matter.

Termination deserves equal attention. A loop should have a maximum time, retry or cost budget; a set of hard failures it may not bypass; and a clear escalation report. Endless retries can burn money while repeatedly making the same incorrect assumption. Stopping with a precise blocker is successful harness behaviour.

Build the Harness From Failures

There is a temptation to design the complete agent platform before delegating the first useful task. I would resist it.

Begin with one bounded workflow and the controls it already needs. Observe where the agent guesses, which commands it misuses, what evidence arrives too late, and which human correction repeats. Then move that correction into the system:

• a missing decision becomes current repository documentation;
• a repeated command error becomes a focused tool;
• a dangerous action becomes a permission boundary;
• a vague failure becomes a better diagnostic;
• an escaped defect becomes a test or runtime check;
• an endless loop becomes a stop condition.

Not every failure deserves permanent machinery. A rare, harmless mistake may be cheaper to correct manually. A specialised platform can be worthwhile when many teams share the same governance or observability problem, but adopting one introduces another dependency, another policy language and another place where state can disagree.

The phrase “the harness is the product” is deliberately forceful, but it should not be read as “the model is irrelevant”. Better models expand the work that can fit inside the boundary. They may need different prompts, tools or retry strategies. The harness is the durable engineering asset: the accumulated context, authority, state and evidence that turns changing model capability into dependable behaviour.

This is the boundary of working with one agent. The next part of the series introduces several. At that point, every ambiguity here multiplies: two agents can collide on state, duplicate work, consume the same review budget and inherit the same blind spot.

Multiple agents do not remove the need for a harness. They reveal whether we built one.