From Agent Scripts to Orchestration Systems

A shell loop can start an agent, wait for it to finish and start the next one. It may be the correct system for months.

Then the process dies after the agent has opened a pull request but before the loop records completion. On restart, should it run the task again? Is the pull request the state, or is a local file the state? Did the agent already update the issue? Which attempt owns the worktree? May another worker claim the task while we investigate?

At that point, adding while true is no longer automation. It is an undocumented answer to a distributed-systems problem.

This is the threshold between an agent script and an orchestration system. It is not a particular number of agents or lines of code. We cross it when work must survive process boundaries, side effects cannot safely be repeated, and operators need to recover without guessing what happened.

Scripts are a good place to begin

There is no shame in a small script. A bounded loop can be easy to read, run and discard:

while task=$(next_ready_task); do
  run_agent "$task" || break
  verify_task "$task" || break
done

For one repository, one operator and work which can be restarted from the beginning, this may be enough. The process is visible, its state is small, and a human can inspect the branch when it stops.

The mistake is either extreme: treating a loop as reliable merely because it has run twice, or adopting a workflow platform before a recurring failure exists. A queue, database and dashboard do not make vague tasks safe. They create more components in which vague tasks can wait.

I would keep the script while these statements remain true:

there is at most one authoritative runner;
unfinished work can be abandoned and restarted cheaply;
tasks have no untracked external side effects;
the operator can understand current state from the repository and process output;
manual recovery is quicker than maintaining automated recovery;
concurrency and resource contention are deliberately small.

When one of those stops being true, identify the missing property. Add that property, not a generic platform shopping list.

The threshold is recovery

The previous chapter described an unattended run as a bounded hand-off. A single run can rely on an isolated branch, fixed checks and a morning review. Several concurrent or recurring runs introduce questions which the individual agent should not answer:

Which work is eligible now?
Who has claimed it, and when does that claim expire?
What should happen after a crash, timeout or changed issue state?
Which retry repeats computation, and which repeats an external effect?
How much capacity and budget may this class of work consume?
What does an operator need to see before intervening?

An orchestrator owns those answers. The model remains responsible for open-ended work inside a task. It should not be the source of truth for scheduling itself.

The distinction matters because model output is probabilistic and conversation state is temporary. If the only record of ownership is “I am working on it” in an agent context, a lost context window has released a lock without telling anyone.

Give every run an explicit state machine

The first durable component is not a message broker. It is a state model.

A minimal task lifecycle might be:

pending -> claimed -> running -> verifying -> review-ready
              |          |           |
              v          v           v
           released   retry-wait    failed
                          |
                          v
                       running

Each transition needs an owner and a predicate. pending -> claimed requires the task to be eligible and capacity to exist. running -> verifying requires an agent attempt to finish, not the whole task to be correct. verifying -> review-ready requires named gates to pass. A timeout goes to retry-wait only if retrying is allowed.

Do not collapse these states into in_progress and done. Operators need to distinguish a live worker from a scheduled retry, and a successful agent exit from accepted work. The earlier chapters’ distinction between local and integrated completion applies here as well.

OpenAI’s Symphony specification is a useful concrete example. It defines a long-running service which reads eligible work from a tracker, creates per-issue workspaces, bounds concurrency, reconciles changed issue state and applies backoff after failures. It also draws an important boundary: a successful run may end in a human-review state rather than Done. Its scheduler state is deliberately in memory: after a restart it polls the tracker and reuses workspaces, without restoring earlier retry timers or running sessions. Recovery does not necessarily mean persisting every internal state.

The implementation details are less important than having one authoritative scheduler state. Workers report outcomes; they do not each edit a shared scheduling file according to their own interpretation. At small scale that authority may be one SQLite database or a directory with atomic file operations. At larger scale it may be a queue and transactional store. The contract matters before the storage product.

Identity makes retries intelligible

Every task and attempt needs a stable identity. Otherwise a retry looks like a new run and duplicate effects are almost impossible to explain.

A useful run record includes:

{
  "task_id": "ORD-184",
  "attempt": 3,
  "state": "retry-wait",
  "base_sha": "4d3c2b1",
  "workspace": "workspaces/ORD-184",
  "started_at": "2026-06-24T01:12:04Z",
  "last_event_at": "2026-06-24T01:18:51Z",
  "failure": "agent process exited during verification",
  "next_retry_at": "2026-06-24T01:23:51Z"
}

The task ID connects attempts. The attempt number makes chronology explicit. The base revision and workspace provide provenance. Timestamps support stall detection. The failure and next action explain why the system is waiting.

This record should be written by the orchestration layer. An agent can supply a hand-off, but it should not decide that its missing process is still alive.

A retry is a business decision

Retrying a text-generation call after a transient timeout costs another request and may return a different answer. Retrying a tool-using agent turn has a larger risk: “create release, notify customers and close issue” may do all three twice.

The orchestrator needs a retry policy per operation or failure class:

which failures are considered transient;
the maximum attempts and elapsed time;
delay and backoff between attempts;
whether the same workspace and agent context are resumed;
which effects must be checked before repeating;
the terminal state when the budget is exhausted.

Backoff protects an unavailable dependency and reduces repeated cost. It does not make an unsafe action idempotent.

Idempotency means that repeating an operation with the same identity has the intended single effect. Before creating a branch or posting a status, the system can look for the task’s existing branch or stable status record; the attempt number belongs in the log rather than the external identity. A migration can record its applied version. Where the external system offers no idempotency mechanism, query and reconcile its state before acting again. If the result remains ambiguous, stop for a decision: a missing response does not establish that the first attempt failed.

There is a small but important ordering problem here. The external effect may succeed and the local state write may fail. No arrangement of two ordinary commands removes that uncertainty. The recovery path must inspect the external source of truth rather than assume the absence of a local acknowledgement means nothing happened.

An agent saying “I probably did not push” is not a recovery protocol.

Recovery requires reconciliation

Persistence alone does not recover a system. After a crash, stored state can be stale.

On startup, the orchestrator should reconcile each non-terminal task against its external facts:

Does the issue or task still exist and remain eligible?
Is the recorded worker process actually alive?
Does the workspace exist, and which revision does it contain?
Did the branch, pull request, deployment or comment already appear?
Are leases or claims stale and safe to release?
Should the task resume, retry from a checkpoint, wait for review or escalate?

The same reconciliation should run periodically because workers can stall without exiting and humans can change issue states through another path. Credentials may also expire while a task is asleep. Before reassigning a timed-out task, stop the previous worker or revoke its ability to write. An expired claim alone does not prevent it from waking up and changing the same resources as its replacement.

There must be an order of authority. The issue tracker may own eligibility, Git may own code provenance, CI may own check results, and the orchestrator may own attempts and leases. Copying all of that into one database creates a useful index, not a new truth. Reconciliation is how the index admits reality changed.

Recovery also needs drills. Kill a worker between an external effect and its acknowledgement, or restart the scheduler with tasks in every state. Then test the less convenient failures: remove a workspace, make the tracker unavailable and exhaust the retry budget. If the only tested path is uninterrupted success, the durable system is durable by assertion.

Routing should express eligibility and pressure

Once work is durable, it still should not all run at once. Routing decides which worker may take which task; backpressure decides how much work may enter the system.

Eligibility should include dependencies, required capabilities, permissions and current repository state. A documentation task does not need production credentials, while UI verification may need a browser-enabled environment. Unresolved schema decisions should keep dependent work blocked even when an agent is idle.

Routing can begin with explicit labels or a small rule table. There is no need to ask another model which model should handle every task if requires-browser and read-only answer the question. A learned or model-based router becomes worthwhile when the rules are genuinely ambiguous and mistakes can be measured.

Backpressure needs limits at several levels:

total concurrent agents;
concurrent agents per repository or environment;
tasks waiting for human review;
token or monetary budget per project;
expensive shared checks such as end-to-end suites;
retries allowed to compete with new work.

The review queue is especially easy to omit. If ten tasks are review-ready and nobody can inspect them, spawning an eleventh worker is not throughput. It is inventory.

Fairness is also a policy. Strict priority can starve low-priority maintenance forever. First-in-first-out can let one large blocked task occupy scarce capacity. The correct rule depends on the work, but it should be visible and testable rather than emerging from whichever agent polls fastest.

Observability is evidence, not animation

Several terminal panes make activity visible. They do not explain the system after a restart.

Operational observability should answer:

what is running, waiting, retrying, blocked or ready for review;
which task, attempt, revision, workspace and agent are involved;
when useful progress last occurred;
which validations ran and what they established;
which external effects occurred;
how much time and model budget the run consumed;
why the orchestrator chose its next transition.

Emit structured events for state changes and consequential actions. Keep stable identifiers in every event. Metrics can show queue age, attempt counts, completion latency, failure classes and review backlog. Traces can connect an agent turn to the tools and gates it invoked.

Do not confuse more logs with better observability. Raw model transcripts are expensive to search, may contain sensitive data, and often fail to state the scheduling decision an operator cares about. Preserve them when necessary for forensics, with appropriate controls. Keep the run ledger concise enough to use during an incident.

The Utah agent harness demonstrates the value of composing an agent loop from durable steps and events. Its LLM calls and tool executions are independently recorded and retryable, while separate functions handle acknowledgement, replies and failures. That design does introduce an external execution service and its operating model. The point is not that every loop needs Inngest; it is that a process boundary should not erase already completed steps.

Separate policy from execution

Agent orchestration mixes two things which change at different rates.

Policy describes eligible tasks, permissions, validation gates, budgets, stop conditions and the required human hand-off. It belongs near the repository and should be reviewed with it.

Execution provides scheduling, claims, timeouts, retries, workspace lifecycle, event delivery and logs. It should not need custom code for every repository rule.

This separation lets a team change review-ready criteria without rewriting the scheduler. It also stops an orchestration upgrade from silently changing what the repository considers acceptable.

Be careful with configuration which is merely executable code wearing YAML. If a workflow file can run arbitrary shell hooks with broad credentials, it is trusted code. Version it, review it and restrict its environment accordingly.

The operational bill is real

A durable orchestrator replaces manual uncertainty with machinery. The machinery has a cost:

a state store needs migrations, backup and corruption handling;
queues introduce duplicate delivery, stale leases and ordering questions;
workers need versioning, credential distribution and isolation;
retries can amplify an outage and the model bill;
dashboards and alerts require ownership;
repository policy can drift from platform configuration;
upgrades may change resumption or tool behaviour;
the orchestrator becomes another privileged route into source control and external systems.

Centralising coordination makes policy and recovery easier to inspect, but creates a scheduler whose failure can stop the fleet. Distributing coordination can reduce that bottleneck, but makes ownership, consensus and debugging harder. For most small teams, one modest authoritative scheduler with independently recoverable workspaces is a sensible trade-off. Designing a highly available agent control plane before the first restart failure would be difficult to justify.

Buying a workflow service transfers some implementation work, not responsibility. You still define idempotency, authority, terminal states and the meaning of a retry. Building your own keeps the mechanism close and can start small, but every special case becomes yours to operate.

Grow from the failures you can name

The path from script to system can be incremental:

give every task and attempt a stable identity;
persist a small explicit state machine;
make one task execution safe to repeat or reconcile;
add bounded retries for named transient failures;
enforce concurrency and review-queue limits;
emit structured transition and validation events;
test restart recovery and stale-work reconciliation;
add specialised routing only when one worker class no longer fits.

Stop when the observed reliability is sufficient. A SQLite file, a repository-owned policy document and good process supervision may be the finished architecture. The goal is not to qualify as an orchestration platform. It is to make unattended work recoverable and its claims reviewable.

Once this machinery works, a new bottleneck appears. The system can schedule more tasks, retain more state and return more candidate changes than a person can responsibly absorb. Orchestration moves the queue; it does not abolish judgement.

That is the bridge to Part IV of this series, beginning with the human coordination ceiling.