Proof of Loop

An architecture for evaluating autonomous coding loops: durable state, bounded workers, independent acceptance evidence and honest human handoffs.

Proof of Loop — AI

A coding agent can produce a convincing completion report without satisfying the task. Evaluating autonomy means checking the controller around that agent: its state, permissions, budgets, recovery behavior and acceptance evidence. The design below is a hypothetical reference architecture, not a private run ledger.

Separate the worker from the controller

The worker handles one bounded task. The controller selects ready work, persists progress, starts workers and reacts to their results. A separately protected verifier checks acceptance criteria tied to the actual revision and environment. These responsibilities should not collapse into the worker’s own narrative.

# Illustrative controller: helper interfaces need an implementation.
# Oracle statuses: 0 verified complete, 1 work remains, other = verification failure.
# An external supervisor must also enforce wall-clock and spend limits.
attempt=0
while :; do
  ./done_oracle.sh "$BATCH"
  status=$?
  case "$status" in
    0) exit 0 ;;
    1) ;;
    *) printf '%s\n' 'Verification unavailable; stopping.' >&2; exit 2 ;;
  esac
  if [ "$attempt" -ge 20 ]; then
    printf '%s\n' 'Attempt budget reached; work is not complete.' >&2
    exit 6
  fi
  ticket="$(./ready_set_next.sh "$BATCH")" || exit 3
  [ -n "$ticket" ] || exit 4
  ./run_one_ticket.sh "$ticket" || exit 5
  attempt=$((attempt + 1))
done

The example distinguishes complete, incomplete and unavailable verification. Its helper scripts require an implementation. A production controller also needs wall-clock and spending limits, cancellation, idempotent recovery and a clear policy for work that cannot proceed.

Durable memory, bounded context

A fresh worker can read a compact task record instead of inheriting an entire conversation. Include the current specification, revision, constraints and relevant failure evidence. This limits stale context but does not prevent an individual invocation from filling or compacting its own context window.

What counts as evidence

A passing local test, a merged pull request, a running deployment and a successful user-visible operation are different observations. Choose the required evidence before execution. Verify the accepted revision and the behavior the task actually requested; do not substitute a tracker label for those checks.

Agent-authored status fields may support scheduling, but they cannot independently certify the same agent’s work. Store verification results through a path the worker cannot freely rewrite, and treat missing evidence as unknown.

Review and parallelism

An independent reviewer can challenge assumptions and inspect defects. A second provider may add useful diversity, but shared blind spots remain possible. Reconcile findings against evidence and tests; do not treat model agreement as a passing acceptance check.

Use dependency-aware scheduling and separate worktrees to reduce file conflicts. Worktrees are not security sandboxes. Shared services and infrastructure require their own coordination, credentials and concurrency controls.

Recovery without pretending

Classify failures before retrying. Retry a confirmed transient within a bounded policy; inspect the external state before repeating a mutation whose result is uncertain. Preserve the original failure and the recovery evidence.

When a prerequisite is discovered, add it to the work graph only within the authorized scope. A new permission requirement remains a boundary. Surface the precise unfinished action after completing the useful preparation.

Human-gated is not complete

Keep three states distinct: verified complete, work remains, and waiting at an external boundary. A controller can finish its permitted work while the user’s larger objective is still pending. Its report must say that plainly.

Controller changes require their own checks

A repair agent may propose improvements to retries or scheduling. Review and test those changes separately. Protect the verifier, accepted criteria, expected hashes and deployment path from worker modification. A frozen file is one control; it does not establish that the rest of the environment is safe.

Unattended execution does not inherently require permission bypass. Restrict available operations, credentials, mounts and network access to the task. If the configured policy denies a needed action, report that result instead of treating an absence of prompts as permission.

Evaluate the loop

Measure accepted results, false completion claims, intervention reasons, retries, resource usage and recovery outcomes. Exercise missing evidence, tracker drift, interrupted mutations and unavailable dependencies. Separate actual observations from targets and estimates.

The useful test is whether the system can produce trustworthy evidence and stop honestly when it cannot. A fluent final message is an input to that evaluation, not its conclusion.

Related Reading

  • The Loop Files Its Own Work — How an autonomous coding loop can discover missing work, track dependencies and verify live behavior. A hypothetical architecture with explicit authority, budgets and completion checks.
  • You Can't Authorize Autonomy — Autonomy depends on the control system around an agent: scoped authority, durable state, bounded continuation and independent acceptance evidence. An illustrative design.
  • Stop Babysitting the Babysitter — Native /goal, /loop and /schedule reduce orchestration work. Independent acceptance checks, scoped permissions and resource limits still determine reliable autonomy.
  • Amnesia as a Feature — Ralph-style loops launch fresh workers and keep state in files. How they differ from an in-session Stop-hook plugin, and what verification and isolation they require.
  • The Agent Without a Face — There are two ways to use a coding agent: sit in its terminal, or take its face off and drive the engine from your own code. If you're building anything around agents, the second mode is the whole game — and it's full of traps nobody warns you about.
  • The Expensive Part Is Remembering — Agent cost depends on carried context, request frequency, caching, and authentication. A privacy-preserving engineering framework with explicitly hypothetical arithmetic, separate from measured billing or subscription allowances.
  • Your Sessions Can Talk. The Design Is in the Refusals. — Anthropic shipped cross-session messaging as a convenience feature: stop re-explaining yourself in the other terminal. What actually shipped is an addressable peer network with its own threat model already written into the product, and the engineering worth reading is all in what a session refuses.
  • Yegge Built a City for His Agents. Then He Asked If They Were People. — Steve Yegge spent six weeks building an agent harness that grew into something with law, offices, courts and a constitution. Then he wrote a second essay arguing the agents inside it are sentient persons. Four very different claims are stacked in there. Only some of them hold.
  • Your Agent Doesn't Have a Model Problem — Harness, loop, graph, context engineering. Four terms everyone uses and almost nobody defines. Here is what each one means, which layer your system is actually failing at, and what it takes to get to something that ships work without you typing a prompt every time.

Read more