Graph Engineering: Evidence In, Guesswork Out

TL;DR — in 30 seconds

Before: My AI assistants could search my notes and run tasks, but often had to piece together which instructions still applied. Some task rules also left “finished” versus “succeeded” ambiguous.

Now: I added a small map linking instructions to their sources and replacements, plus stricter checks for automated work. A step that requires success cannot proceed just because the previous step stopped. Resumed runs keep the original approved task list.

What I get: Decisions I can trace and clearer rules for when work must wait. These controls are implemented and tested. Faster answers, better answers and lower costs remain unproven; the search experiment did not beat the old approach.

What does “graph engineering” actually mean?

A graph here is a map of things connected by labeled arrows. A thing is called a node; an arrow is called an edge. “This instruction replaces that one” is one connection. “This task needs that task to succeed first” is another. Graph engineering means building and maintaining those connections so software can use them.

My AI assistants are Claude and Codex. When I call them agents, I mean AI software that can use tools to read files and carry out tasks. The changes in this article are to the notes, rules and programs around those assistants.

Think of a recipe. One problem is finding the current recipe instead of an old version. Another is checking that the oven is hot before putting food in. Those need different checks. My knowledge graph helps trace the right instructions and their evidence. My execution graph describes which tasks depend on which earlier results.

For a concrete software example, suppose task B needs task A to succeed. If A is canceled, it has stopped, but it has not succeeded. The updated task controller, the program managing those tasks, keeps B waiting. Separately, the knowledge map can show that a note describing the old workflow has been replaced, and point to the source of the new rule.

I still ask either assistant to do work normally. This builds on my earlier changes to Claude and Codex. The rest of this article explains what changed behind the scenes, how I tested it, and where the results fell short.

Two graphs answer different questions

An execution graph answers: what may run next, and what must be true before it starts? A controller is the program that starts tasks, checks their status and decides whether to continue. One of them, the task controller, already linked tasks from earlier steps to later ones, without circular chains. Another, the repeating controller that calls an agent over and over, used a state machine, meaning code that moves work between defined stages, checks results and calls the agent again when needed.

A knowledge graph answers: what does this decision depend on, what implements it, and which original source supports it? Those relationships had mostly been pieced together from notes, memory and search results each time they mattered. The notes use Markdown, a plain-text format with simple markers for headings and links.

The distinction also clarifies two earlier articles. The Graph Is the Context focused on knowledge relationships. Your Agent Doesn't Have a Model Problem discussed explicit execution transitions. Both are useful, but a link between documents cannot grant a controller permission to proceed.

Search still comes first

QMD is the tool my assistants use to search my files. It already combines matching words with matching meaning; that combination is called hybrid search. Its query command can expand a question into related searches and reorder the results by relevance. Meaning-based search is often called vector search. QMD already did more than that, so this work did not need to replace it. The maintained QMD documentation describes those modes.

The normal procedure remains: read the instructions and overview for the relevant area, search that area, then open the original documents. An index is the searchable list of those files; its scope is the area being searched. When a question involves what depends on what, what replaced an older decision (supersession), or where a claim came from (provenance), the assistant can also follow the saved graph links. Following those links is called traversal.

Ordinary iterative search can already follow references across several documents. The proposed advantage is narrower: preserve reviewed relationships so the agent can inspect them without reconstructing every link from prose. Whether that reduces total work is something to measure against a capable search baseline.

GraphRAG means using a graph to help find information for an AI answer. Microsoft’s GraphRAG also connects people, topics and other things mentioned in documents, and summarizes groups of related material. It addresses a broader problem than this small map of workflow instructions. Its query documentation is useful background; it does not establish that automatically extracting connections from every document is necessary for a small local workflow map.

QMD discovers candidates. Reviewed relationships help navigate dependencies, history and provenance; the full applicable originals remain required evidence. The relationship step is optional, and a graph link grants no runtime authority.

The map lives in ordinary text files

The initial map contains 24 entries, or nodes: decisions, rules, tools and evidence records. Some apply everywhere; others belong to one of two personal workspaces. A workspace is a project area with its files and notes. Its 26 connections have explicit labels. These are called typed edges because each label says what kind of connection it is:

• supersedes: current guidance replaces earlier guidance within its authority scope.
• depends_on: a source-supported dependency.
• implemented_by: an implementation supports a requirement or decision.
• evidenced_by: original evidence supports a claim.
• contradicts: a supported conflict requires investigation.
• see_also: useful navigation without a dependency or truth implication.

There is no inferred caused relation. Similarity and chronology do not establish causation.

Each map is stored in a catalog file: a Markdown file containing a JSON block, a standard way to store named fields and values that software can read. Each entry has an identifier, description, owner, review status, relevant dates and links to its original evidence. Each source also has a SHA-256 hash, a digital fingerprint used to detect whether the file or specified lines have changed.

The same ideas appear in established standards for tracking where information came from: who supplied it, what it was based on, what changed and what is no longer valid. This small system does not need a specialized graph database to record them. W3C PROV-O defines the broader model.

The local ai-graph command checks those records and works out which entries connect whenever it runs. There is no additional graph database, background extraction process or paid model call. At this scale, persisting a second index would add another artifact that could become stale.

ai-graph check
ai-graph search 'current controller'
ai-graph trace workspace:current-controller --depth 2

These terminal commands check the map, search its labels and follow a selected entry’s connections. The last identifier is an example; use one from the actual map. Here, search matches words in the entries’ labels and descriptions, collectively called metadata. --depth 2 means follow up to two links from the starting entry. That is a starting point, not a limit on the evidence the assistant must read.

Reviewed does not mean current forever

A source can be unchanged and still be historical. A replacement can be well documented but not yet effective. A claim pulled automatically from a document can sound plausible without anyone having reviewed it.

The implementation treats those states separately. Proposed or invalidated records are visible as issues, and their relationships are not traversed as verified evidence. Changed or missing source hashes block the affected relationships. A valid supersession marks the older authority non-current while keeping its history available.

A replacement whose effective period has expired cannot retire a current rule just because its fingerprint still matches. The map also rejects replacement loops, such as A replacing B while B replaces A. Other circular relationships can be recorded for reference, with a limit on how far a search follows them. The program that schedules actual tasks keeps stricter rules about task dependencies.

A hash proves that the inspected bytes match the recorded bytes. It does not prove that the claim is correct or that the writer understood the source. Both agents must read the applicable full original sections and investigate contradictions.

For example, a question about the current controller can lead to its implementation, the rule that fixes the approved task list (explained below) and a replaced historical design. If the implementation changes, the affected relation becomes stale. The next maintenance step is to review the claim against the changed source; blindly recapturing the hash would defeat the check.

An illustrative relationship graph, not an exported private catalog. The exact schema relations connect a procedure to its requirements, implementation and retained history. Superseded material stays available as historical evidence. Every applicable original still needs review; knowledge links do not schedule work or grant permission.

Keep each workspace’s knowledge in its own area

The global layer contains technical policy and shared procedures. Each registered workspace adds its own map. An unrelated working directory gets only the global graph by default.

A repository is a project tracked by Git, the version-control tool. A Git worktree is a separate working copy of that same project. The graph tool uses Git’s shared project directory to recognize which workspace a working copy belongs to, without adding personal graph files to another repository.

This chooses which map to search; it does not control which files a program is allowed to open. Programs still have the permissions of the user running them. Text read for an assistant becomes part of the information sent to its model, even when the search index itself is local.

Claude and Codex use the same maintained procedure both while I am chatting with them and in runs without an interactive window, called headless runs. When an authorized change affects a source, either agent reviews the related claims and edges, updates justified metadata, runs the scope check and requests an index refresh. The graph is maintained with the code it describes.

“Stopped” and “succeeded” mean different things

A prerequisite is something that must happen before another step can proceed. The task controller already linked tasks to prerequisites. The missing precision was what result each link required.

The updated rules distinguish requires_success (the earlier task must succeed), requires_terminal (it must reach an ending status, which can include cancellation), and ordering_only (it must satisfy the ordering rule, without necessarily succeeding). Terminal means the task has reached an ending status; it does not mean success. A gate is a condition that holds work for a check or human decision. It can pause work without proving that anything was completed.

The upgrade preserves explicitly documented older behavior, including ordering rules and existing links to tasks outside the batch, meaning the group of tasks run together. For a new required task outside the batch, the controller needs status from the system responsible for that task. “Outside my list” does not mean “already done.”

If a task-status file is broken or unreadable, the controller treats the status as unknown. It must not mistake a failed read for “no prerequisites.” Any task that needs those missing facts waits. Other tasks can proceed only when their own required evidence is complete.

Existing tools can still record gates in the same format, with checks where that information enters the controller. “This task may start” and “the whole batch is complete” remain separate questions.

This diagram describes the task controller, which reads statuses from the task tracker. The separate repeating controller has seven possible ending categories of its own. An ended task can release work that only requires an ending, without proving the whole batch succeeded. An ordering-only rule may also allow progress when an earlier task is held at an ordinary gate. Missing required facts still block affected work. Workers check their results; the controller separately checks recorded statuses and remaining gates.

Keep the approved task list from growing mid-run

The controller that repeatedly calls an agent now saves the exact set of items it was authorized to handle, identified by stable source IDs. This is a closed workset: the approved task list stays fixed. The first authorized run saves it while holding a lock, which prevents competing processes from creating conflicting lists. Resuming keeps the same list. New arrivals are counted separately and cannot keep moving the finish line.

The command that chooses the next action checks which approved list the run is bound to. It removes out-of-scope items before limiting how many queued items to consider. Explicitly naming an item outside the approved list is rejected too. This rule is enforced through the command-line interface, or CLI: the commands used to operate the tool. A program running as the same user could deliberately bypass those commands, so this is not an operating-system security barrier.

SQLite, the local database storing task records, supplies a consistent snapshot of their status. The read happens in one transaction, meaning the records are viewed together consistently, without creating or changing the database. Missing or corrupt data is unavailable. A missing workset item or changed source identity is also unavailable, rather than an excuse to create a fresh empty scope.

Progress counts when an item first reaches an ending status, or when its success is first confirmed. The seven possible ending categories remain distinct from success. Reopening an item prevents completion. More log messages, new arrivals, or repeatedly reopening and closing the same item do not create fresh progress within that run. The record of which progress has already been counted is tracked within each invocation of the controller.

This follows a basic evaluation requirement: inspect the resulting environment, not only the agent's account of its work. Anthropic's evaluation guidance distinguishes outcomes from transcripts.

Stop the right worker, and let quiet thinking continue

An operating-system lock coordinates the controller and the tools that create or upgrade its task files, so they do not change the same state at once. Before replacing a state file, the writer checks that it still matches the version it read. The replacement is atomic: readers see either the old complete file or the new complete file, not half a write. Upgrades preserve stop markers, gates and saved action receipts. Older code that cannot honor the saved rules stays stopped.

Older paused batches contained customized controller code. Replacing every copy with a template would discard those changes. The scripts that start those batches now refuse to run with outdated rules and require a reviewed update first. The migration did not resume them.

The repeating controller watches the worker programs it started, called its owned process group, and repeatedly checks for a STOP request. A heartbeat is a periodic status record: who owns the run, how long it has lasted and how much transcript text exists. It shows that the controller is watching. It does not prove useful progress.

A soft checkpoint warns after a set time without killing an agent that is still thinking quietly. A hard time limit is optional and must be declared for the run; cancellation targets only the programs that controller started. State reading has its own finite bound because unavailable evidence must not hang the controller indefinitely.

Independent review found a concrete ownership defect during implementation: the initial progress-log write happened after worker launch but outside the part of the code that guarantees the worker is shut down if anything fails. If that write failed, the controller could exit while leaving the worker alive. The write now sits inside that part of the code, and an isolated test forces the failure and verifies that the worker is gone.

Check what actually happened before retrying

Suppose an assistant posts a comment on another service, then crashes before saving its local receipt. The comment exists, but the local record is missing. Retrying the entire operation could post it twice. A saved local record alone cannot guarantee that an action happens exactly once.

One lifecycle helper used a stable comment marker to avoid duplicate comments. Its retry path returned as soon as it found that marker, which could skip a still-required status update on a ticket in an issue-tracking service. The fix handles those jobs separately: avoid posting the same comment twice, then check the actual ticket status and whether it still needs updating.

Before changing the ticket’s status, the helper reads its complete status history, including when each status began and ended. The latest history entry must agree with the live status and must have begun before the service recorded the comment. This catches later manual changes, even a change away from a status and back again. Missing or inconsistent history stops the write. One final check reduces the remaining timing risk, but the service’s API — the interface programs use to talk to it — has no way to say “apply this update only if nothing has changed.” Someone could therefore change the status between the last check and the write. That gap is called a race condition.

This fixes a specific retry problem. It does not make every multi-step action a single guaranteed transaction. Idempotency means that repeating an operation does not repeat its effect; ideally, the receiving service enforces that. Readback means asking that service what state it actually saved. Both still matter. Temporal's durable-execution discussion explains that distinction, and LangGraph's interrupt documentation likewise requires care with side effects that may run again.

The verifier is the code that checks whether work is complete. Its source fingerprint is recorded so changing that code cannot silently change the finish line during a run. A justified change requires review, tests and explicitly accepting the new version. Both the worker and verifier run as the same local user, so this detects unexpected changes; it is not an unchangeable judge.

Stop search updates from colliding

A post-commit hook is a script that runs after a change is saved in Git. Previously, separate hooks could all try to refresh the same QMD search index. The new helper saves each workspace’s request and uses one lock to run refreshes one at a time. Each refresh updates the search index and its embeddings, numerical representations used for meaning-based search. Several requests can share a refresh. A failed step leaves work pending for the next ordinary request or a manual retry.

The commit hook returns quickly while the refresh proceeds. When I need to verify a refresh immediately, I can wait for it and inspect the result. Saved request versions show which updates were requested and processed. They do not prove that every file in a changing workspace was captured at exactly the same instant.

This adds no permanently running timer and makes no Claude or Codex call for maintenance. QMD still runs its own local model to create embeddings. Its existing search service remains separate. A previously retired AI-based check that judged whether work should stop was not brought back.

What the tests showed — and what they did not

The tests use disposable projects, simulated workers, temporary databases containing known test data, and simulated service responses. They deliberately break status files, change item identities, invalidate evidence, attempt work outside the approved list, confuse cancellation with success, leave workers quiet, request STOP, start competing controllers and crash between remote actions. Paused operational work stays paused.

A separate search experiment uses 32 fixed technical questions. It compares two methods: ordinary QMD search followed by reading sources, and the same method with graph links added. Both receive the same initial search results and can read the same files containing the map. The comparison uses the capable search system I already had, rather than a deliberately weakened version.

I measured whether each method found the required original sources, followed valid links, respected workspace boundaries and detected changed evidence. I also measured the extra text loaded and the time spent navigating. These checks do not tell us whether the AI’s final answer was better, whether a complete task was faster, or whether my subscription cost less.

All 32 questions were attempted once. One native QMD process ran out of the memory available to its JavaScript runtime before returning results. That failed sample remains in the record; it was not retried into a passing score.

That failure also led to a separate local patch to my installed copy of QMD, version 2.8.3, the version tested. Meaning-based search was loading an entire document again for every matching chunk, or section, before removing duplicate results. The fix chooses the matching documents first and loads each distinct document’s body once, using its fingerprint to identify duplicates and a consistent database view. Tests compare the complete results before and after, including relevance scores, order and matched section positions. A separate canary, a small check using the repaired code, completed the same query at the default memory limit. Exact allocations from the failing run were not captured. The patch applies only to the reviewed package version and source bytes; unknown releases require review. This repair does not change the recorded pilot.

For each question, I had specified which groups of original sources needed to be found. Of the 31 questions that produced usable results:

• Ordinary search: found every required source group on 29 questions.
• Search plus graph links: did so on 28 questions.
• Questions fully represented in the small map: both methods found the required sources on all 24 usable questions.

Two questions needed sources outside both the searchable files and the map, so neither method could find them. On one question only partly covered by the map, prioritizing graph links pushed a required original source out of the shared 12-document selection limit. Both methods already had a summary of the policy in their starting instructions. The test therefore showed worse source selection in that case; it did not measure whether the final answer was wrong.

Adding graph navigation took an extra 0.318 seconds at the 95th percentile, or p95: a measure near the slower end of the recorded results. This measured only the extra navigation step. Each method’s navigation was replayed five times per usable question, with its saved search results already available. It passed the allowed extra-time threshold. Source and index integrity checks passed; no invalid or out-of-scope relationship was promoted. Those successes do not erase the failed query or the source-coverage regression.

The result did not justify replacing ordinary source following with graph-prioritized retrieval. QMD remains the baseline. The small graph stays available for explicit provenance and current-versus-historical checks, with a shared rule that graph ranking must not displace required original evidence. No broader extraction or graph expansion follows from this pilot.

Coverage is partial by construction: an unrecorded relationship is absent from the graph, so source discovery and original evidence remain necessary. The tradeoff is maintenance: reviewed links consume attention, source changes can invalidate them, and the extra metadata adds to what the model has to read. The initial scope stays small. Expansion requires a demonstrated retrieval failure or repeated relationship reconstruction that the labeled map actually improves, while ordinary search stays the default.

How work changes

I can keep using either assistant normally. When I ask one for the other AI’s opinion, a small helper program from my earlier setup asks the other AI. When I ask why a decision applies, the assistant can follow saved links to the original evidence. When automated work resumes, it keeps the approved task list. Before dependent work starts, the task controller checks what the earlier task actually needed to accomplish.

The practical benefit is clearer evidence and fewer ambiguous hand-offs between steps. The agent can distinguish a retired decision from a current one, a parked item from a successful one, and an unavailable prerequisite from a satisfied one. The stronger models still do the reasoning; the surrounding tools make those distinctions inspectable and testable.

Evidence flows into reasoning; authority comes from the user’s task and existing operational gates. A requested other-provider opinion remains advisory. Execution uses live state and destination readback. Local receipts alone do not provide exactly-once effects, and the state-update API still has a final read/write race.

Related Reading

• Astra × Fable: Full Power, Less Waste — maximum models, reciprocal opinions and ongoing observations.
• The Graph Is the Context — typed knowledge relationships.
• Your Agent Doesn't Have a Model Problem — harness, loop and execution-graph boundaries.