An agent can cause a security incident without producing a single bad line of code.
Give it a shell, your home directory and a credential-rich environment, then ask it to investigate an untrusted log. A prompt injection in that log can steer the agent towards files that have nothing to do with the task. Even without an attacker, an over-broad command can modify the wrong directory or send data to an unexpected service.
The reverse is also possible. The agent can remain perfectly contained while producing a subtle authentication flaw that later reaches production.
Those are different failures. The first concerns what the agent can reach while it works. The second concerns the software it leaves behind. Calling both of them “AI security” is accurate but not very useful; the controls are not interchangeable.
I find it clearer to draw two boundaries:
- The runtime boundary limits the agent, its tools and the programs it executes.
- The artefact boundary determines what evidence generated code must provide before we trust it.
A sandbox helps with the first. It proves almost nothing about the second.
Boundary One: Contain the Runtime
Coding agents are powerful because they inherit ordinary developer capabilities: reading files, running shell commands, installing dependencies, calling APIs and changing repositories. That also means they inherit our blast radius, unless we deliberately reduce it.
The basic rule is unsurprising: if a task does not require a capability, the agent should not have it.
For a local coding session that usually means:
- work inside a disposable container or VM
- mount only the repository and other explicitly required paths
- keep production data and credentials outside the environment
- use task-scoped, short-lived credentials where authentication is necessary
- restrict network egress to the services the task needs
- expose a small tool set rather than every integration available to the account
- record tool calls and external changes so the session can be reconstructed
“Use a container” is not a complete security design. A privileged container with the Docker socket, a writable home-directory mount and unrestricted egress is mostly a tidy way to retain the original problem. Isolation is about reachable capabilities, not the logo on the runtime.
Claude Code illustrates why permissions and isolation are separate layers. Its permissions documentation describes rules enforced by the agent harness on tool use, while sandboxing adds operating-system enforcement around shell processes. Anthropic recommends using both. It also recommends bypassPermissions only inside an isolated environment.
This is defence in depth, not mathematical certainty. A deny list can omit a sensitive path. An allowed command can have surprising behaviour. Network access can turn an innocent read into exfiltration. The sandbox needs a narrow filesystem, a narrow network and credentials with a narrow purpose.
The Agent and Its Programs Are Not the Same Actor
There is another distinction hidden inside the runtime boundary. The agent harness may need a model credential and access to orchestration services. Code generated by the model rarely needs those same privileges.
Malte Ubl and Harpreet Arora’s four-actor model separates the agent, its secrets, generated-code execution and the filesystem. The useful part is not the number four. It is the refusal to run everything inside one trust domain.
Suppose an agent needs to call an internal API while testing a change. Putting the raw token in its environment lets every generated subprocess read that token. A stronger design keeps secrets outside the code-execution sandbox and injects authority only into approved requests. The program can perform the permitted operation without learning a reusable credential. The proxy must still restrict what those requests can do; hiding the token does not prevent misuse of the API during the session.
This design has a cost. Separate execution environments add start-up time, infrastructure and awkward debugging. Egress proxies need policy and maintenance. Short-lived credentials create another service that can fail. For low-risk, read-only work, that may be unnecessary machinery. For unattended execution or access to customer systems, it is cheap compared with discovering that “the agent had the same access as the engineer” was the entire permission model.
Stripe’s Minions architecture offers a useful production example. Stripe gives each run an isolated development environment, no production access and a curated set of internal tools. Their exact platform is not a blueprint for a small team; the transferable idea is that the architecture removes capabilities before the model has a chance to misuse them.
Boundary Two: Verify the Artefact
Containment answers: what could this session damage? It does not answer: what did this session build?
Generated code can contain the same defects as human code. It can misunderstand a domain invariant, copy an obsolete pattern, introduce a dependency with an unsuitable licence, or implement the happy path while quietly dropping an authorisation check. It may also be difficult for the human reviewer to understand, as discussed in the previous chapter.
The awkward case is when the agent generates both the implementation and the evidence for it. A test suite written from the same mistaken interpretation can agree perfectly with the code. Green is useful; it is not independent.
The artefact boundary therefore starts before generation:
- write acceptance criteria and security invariants independently of the implementation
- derive high-value tests from those criteria, not from the finished diff
- run type checks, static analysis, dependency and secret scanning
- use property tests or fuzzing where many input combinations matter
- review changes to authentication, authorisation, cryptography, persistence and privileges manually
- deploy through a reversible path with monitoring, staged exposure and a tested rollback
Typed code and static tools remove useful classes of error, but they do not prove the business rule. Property tests explore more inputs, but only for the properties we remembered to state. A second model can catch mistakes, but it may share the first model’s blind spot. Each layer contributes evidence. None deserves the word “safe” on its own.
This is where threat modelling remains valuable. Before asking which scanner to install, ask what the component protects, who may call it, what data crosses its boundary and what failure would be intolerable. The answers tell us which evidence matters.
Formal Proof Has a Narrower, Stronger Job
Leonardo de Moura argues that AI-generated software should be treated as a supply-chain input. A compromised model, API or retrieved context could introduce a deliberately subtle defect at scale. His proposed response is an independently stated formal specification and a machine-checked proof.
I agree with the direction, with a qualification. Formal verification does not prove that we specified the right thing. It proves particular properties of a model or implementation under stated assumptions. A verified access-control rule can still encode the wrong policy; a proof about an orchestrator says nothing about the security of the code it orchestrates.
PetriFlow makes that boundary unusually visible. Josh Tuddenham’s example uses a Petri net to prove properties such as termination, a mandatory human-approval path and a bounded iteration budget. For the demonstrated net, the tool enumerates 196 reachable states. That is evidence about that net’s topology. It does not prove that an approver makes a good decision, that a selected tool is benign, or that generated application code meets its security requirements. Tuddenham says so explicitly.
The running system must also honour the model’s assumptions. A termination proof over transitions does not make a tool call that hangs forever return; execution still needs timeouts and failure handling. This is not a weakness in formal methods. It is what honest proof looks like: precise about its subject and its limits.
For a CRUD application, machine-checked proofs may be a poor use of time. For a cryptographic routine, replicated log, financial invariant or permission workflow, they may be exactly the right investment. Between those extremes sit executable specifications, model-based tests, deterministic simulation and property testing. The useful principle is not “formally verify everything”. It is “define correctness somewhere other than the generated implementation”.
Match the Controls to the Risk
Small teams do not need Stripe’s infrastructure to improve either boundary. They do need to stop treating every agent task as equivalent.
A read-only documentation pass over a public repository may need little more than a clean checkout and no credentials. A task that installs unknown dependencies or runs browser automation deserves disposable compute and restricted egress. An agent changing external systems needs scoped identity, an audit trail and explicit approval around irreversible actions. Production access should remain exceptional, short-lived and easy to revoke.
The artefact gates should rise in the same way. A generated internal script and a payment-authorisation change should not pass through the same review path. Higher impact calls for stronger invariants, more independent verification and more human attention.
There is no configuration that makes an agent generally trustworthy. There are only specific capabilities we chose to grant and specific claims we gathered enough evidence to accept.
The runtime boundary asks, “What damage can this agent do now?” The artefact boundary asks, “What evidence lets us ship what it produced?” Keep both questions visible. A sandbox without verification lets us produce unsafe code in a safer room. Verification without containment may eventually give us good code after exposing everything around it.
The next chapter looks at what happens when these controls move into the development loop itself: The SDLC Is Dead: What Replaces It.