Tests Are Backpressure

A coding agent can propose the next change before a slow test suite has finished checking the previous one. That may look productive, but often it is just a queue of unverified work growing faster.

In the previous chapter on declarative development, the difficult question was what counts as correct and who controls that definition. Here I am interested in what happens next. Once those constraints exist, they need to reject bad work quickly enough to affect the agent’s next move. That is the sense in which tests become backpressure.

The Producer Must Feel the Failure

In a streaming system, backpressure prevents a fast producer from overwhelming a slower consumer. The analogy is not exact, but it is useful. An agent produces changes. Compilers, tests, static analysis, runtime checks and reviewers consume them. If generation can continue while verification falls behind, defects accumulate in a batch and become harder to attribute.

Clayton Farr’s Ralph Playbook, a guide to Geoffrey Huntley’s Ralph Wiggum technique, uses the term in this practical sense: tests, type checks, linters and builds reject unacceptable work and direct the agent to correct it before advancing. For that feedback to become binding, the surrounding workflow must enforce the stop; a loop can still ignore a prompt saying that tests must pass.

This rules out a surprisingly common workflow:

1. Ask the agent for a broad feature.
2. Let it modify twenty files.
3. Run the full test suite at the end.
4. Hand a mixed collection of failures back to the agent.

The feedback exists, but it arrives too late. A failing test may relate to the first change, the seventeenth, or an interaction between them. The agent now has to reconstruct decisions made much earlier in its context. So does the human.

A better loop is smaller:

make one coherent change
        ↓
run the cheapest relevant check
        ↓
failure? repair or stop
        ↓
run broader checks
        ↓
record evidence, then continue

The checks in this loop depend on the risk. What matters is keeping each change close to enough evidence that a bad assumption cannot travel far before something rejects it.

Useful Backpressure Has Five Properties

First, it is fast enough to affect the next decision. A formatter that replies in a second can shape every edit. A test suite that replies tomorrow can still protect a release, but it cannot guide an inner implementation loop.

Second, it is close to the failure. “Checkout retry violates idempotency” gives the agent somewhere to look. “Build failed” followed by forty thousand lines of logs is technically feedback and operationally a small punishment.

Third, it is independent of the implementation. If the same agent can weaken an assertion, update a snapshot or replace a dependency with a friendly mock, green proves very little. The right separation depends on risk: visible test diffs may be enough for a small change, while payment, authorisation or migration invariants may need protected acceptance tests owned elsewhere.

Fourth, it is layered. Unit tests are quick and precise but see a narrow world. Integration tests catch incompatible components. Static analysis catches classes of defect that example-based tests may miss. Starting the application discovers configuration and wiring errors. Browser or API exploration finds behaviour nobody thought to encode.

Finally, it is binding. A warning that everybody routinely ignores is decoration. Backpressure needs authority: block the commit, stop the loop, require a review, or make the agent report that it cannot proceed.

The five properties pull against one another. Broad end-to-end tests are often slower and less diagnostic. Very fast unit tests may be tightly coupled to implementation, while protected tests are more independent but harder to debug. They are trade-offs in the design of the feedback system.

Order Checks by Cost and Scope

I would arrange checks by cost and scope. The exact commands vary, but the order is usually something like this:

seconds      formatting, lint, type checking, compilation
minutes      focused unit and component tests
minutes      affected integration and contract tests
longer       service startup, browser/API scenarios, wider regression suite
outside loop protected CI, deployment checks, production signals

Run the cheapest discriminating check first. There is little value in spending fifteen minutes on browser tests when the code does not compile. Equally, a passing type checker does not justify skipping a real interaction with the changed feature.

Selective execution helps, provided the selection itself is trustworthy. A module dependency graph can identify affected tests. A changed-file rule can choose a linter or contract suite. But selection logic becomes part of the safety boundary. If it silently omits a downstream consumer, a fast green result is worse than an honest slow one.

The outer layers should also be independent from the inner loop. Local checks give the agent rapid correction. Protected CI reruns the important checks in a clean environment, using the exact revision under review. Production health may then reveal a latency change under real load or a migration which stalls only against production data.

Disagreement between layers is useful. If focused tests pass but the application cannot start, the test boundary was too narrow. If a local suite passes while CI fails, there may be an undeclared dependency or an environment assumption worth making explicit.

Sometimes that disagreement exposes a defect in the feedback path itself. A useful check cannot affect the next decision if its command is missing from the repository instructions or its failure message hides the invariant being violated. Fixing those problems helps future agents and humans. A one-command test environment can remove repeated setup, while a protected architectural check can turn a recurring review comment into an enforced constraint.

Tests Must Be Allowed to Say No

Agents are very good at removing obstacles. Unfortunately, a test failure can look like an obstacle.

The simple rule “never edit tests” is not workable. Tests can be wrong, requirements change, and a refactor may legitimately replace assertions tied to an old design. The opposite rule, allowing the implementation agent to change any test until everything is green, is worse.

Treat changes to the oracle as a separate decision. Ask why the old expectation is invalid, show the test diff independently, and rerun the new test against the old implementation where practical. A new test that is green before the behaviour exists has not demonstrated much.

This is also where mutation and fault-injection techniques can help. If deliberately breaking the implementation leaves the suite green, the suite has shown us its limit. These techniques cost compute and maintenance, so I would target them at important invariants rather than turn every build into an endurance event.

Flaky tests deserve similar attention. A test that fails one time in twenty trains both humans and agents to retry rather than investigate. Once “rerun it” becomes normal, a genuine intermittent defect can hide in the same noise. Quarantining a flaky test may be necessary to unblock work, but it should create an owned repair task and a visible gap in coverage. Quietly retrying until green destroys the signal.

Execution Is Part of Verification

Automated checks cover behaviour somebody thought to encode. They can still miss an off-screen control or approve a request which the real dependency rejects.

Simon Willison’s agentic manual testing pattern begins with a straightforward rule: do not assume generated code works until it has been executed. For an API, that may mean starting it and exploring the changed endpoint. For a web interface, it may mean driving a real browser and inspecting screenshots. When exploration finds a defect, converting it into a failing automated test preserves the discovery for the next change.

I like this because it avoids a false choice between automated and manual testing. Exploration is good at discovering that our model of the problem was incomplete. Automation is good at making the discovered expectation cheap to repeat. They are two stages of one feedback loop.

The evidence should survive the agent session. Record the command, exit code, relevant output and revision checked, plus screenshots where appearance matters. For example, report that npm test passed on commit abc123 in a clean environment rather than saying only that “tests pass”. Another person then knows what to challenge or reproduce.

Backpressure Has a Cost

There is a cost. Tests must be designed, maintained and sometimes deleted. CI consumes machines and attention. More gates can make a safe change painfully slow, encouraging people to bypass the lot. A team can also optimise what is easy to measure and miss accessibility, operability or whether the feature should exist at all.

Tests will always model only part of the world. They can reject known bad states quickly and repeatedly, leaving people to decide which states matter and investigate risks that the existing system cannot yet express. That is more useful than treating a green suite as permission to ship or asking a human to reread every generated line.

Backpressure makes mistakes cheaper when the refusal arrives early and cannot be ignored. It does not remove judgement; it gives us something better to spend judgement on.