Agents Are Programs Too

Chapter 14 held the corpus, program, model, optimizer configurations and paired harness fixed, changed only the objective, and closed the editorial task. The prediction it left behind was that a reflective optimizer improves in proportion to how much its feedback can say — and that a task whose failures carry a test name, an expected value and a traceback should therefore be the book’s best case.

To measure that we need a task with those properties. Repository repair has them. But it introduces a limitation the editorial task never had.

Our editorial program receives a sentence and its local context. Both fit in a prompt. A repository does not. You cannot hand a program the whole environment and expect it to behave, and you cannot hand it a curated slice without having already made the decision you wanted the program to make.

So the program needs to reach into the environment itself. That is what this chapter adds, and it introduces a new class of failure alongside the ones we already know how to name.

    flowchart LR
    B[tool boundary unsafe] --> B2[program obtains evidence it should never have had]
    R[reasoning over safe evidence wrong] --> R2[program obtains correct evidence, draws the wrong conclusion]
  

These are separate failures with separate remedies, and the chapter’s measured result is that they come apart cleanly. The boundary held. The evidence was correct and complete. The answer was still wrong.

1. Access, not intelligence

The naive shape is a bigger prompt; the stronger shape gives the program a way to ask:

    flowchart LR
    subgraph naive
        I1[issue + huge repository context] --> MD[model] --> PT[patch]
    end
    subgraph agent
        I2[issue] --> AG[agent]
        AG --> SR[search_repository]
        AG --> RF[read_file]
        AG --> IS[inspect_symbol]
        SR --> DG[diagnosis]
        RF --> DG
        IS --> DG
    end
  

The important thing about the second shape is what it does not change. A tool-using agent is still a language-model program. It has a signature. It has dependencies. It produces evidence. It can be evaluated, optimized, versioned, promoted and rejected under exactly the machinery the previous fourteen chapters built. “Agent” is not a new category of software. It is a program that receives ordinary call-time inputs and can acquire additional evidence while it runs.

That reframing is the reason this chapter sits where it does. Nothing about contracts, metrics, splits or promotion becomes optional because the program can now call a function.

What changes is that the program’s complete evidence state is no longer knowable before execution. The initial request can still be fingerprinted. Tool observations cannot: they are produced by the trajectory and become part of the model’s effective input as the run proceeds. The run artifact therefore has to record and fingerprint the acquired evidence as well as the initial request. Everything difficult in this chapter follows from that one fact.

2. Deriving a boundary instead of listing safe tools

The instinct is to write three read-only functions and call the job done. Here is a version of that instinct, close to what most tutorials show:

from dataclasses import dataclass
from pathlib import Path

@dataclass(frozen=True)
class SearchHit:
    path: str
    line: int
    text: str

def search_repository(root: str, query: str) -> list[SearchHit]:
    hits = []
    for path in Path(root).rglob("*.py"):
        for line_no, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
            if query.lower() in line.lower():
                hits.append(SearchHit(str(path), line_no, line.strip()))
    return hits[:20]

def read_file(root: str, path: str, start_line: int = 1, end_line: int = 80) -> str:
    target = (Path(root) / path).resolve()
    root_path = Path(root).resolve()
    if root_path not in target.parents and target != root_path:
        raise ValueError("path escapes repository root")
    lines = target.read_text(encoding="utf-8").splitlines()
    return "\n".join(lines[start_line - 1 : end_line])

Read-only, path-checked, capped at twenty hits. It looks defensible. It is not, and the ways it fails are more instructive than the ways it works.

root is a parameter. This is the load-bearing defect and it is invisible because it looks like good style. If root reaches the tool from anywhere the model can influence, the containment check compares the target against a root the model chose, and it will pass. A boundary whose reference point is an argument is not a boundary. The root must be closed over in application code and must never appear in a signature the language model can populate.

The search reads every file completely. On the fixture this is instant. On a real checkout it walks .venv, node_modules and .git, loads each file into memory, and dies on the first byte sequence that is not valid UTF-8. A tool that crashes returns its traceback into the agent’s observation stream, where it becomes context the model will try to reason about.

Nothing bounds the response. end_line has a default, not a limit. An agent that asks for lines 1 to 10,000,000 gets them. Tool output that overwhelms the model is not a performance problem; it is a correctness problem, because the relevant evidence is now buried in irrelevant evidence the program itself requested.

The exclusion set is empty. Everything inside the root is readable, including .env, credentials fixtures, and anything else a repository happens to contain. Containment answers “is this inside the repository”. It does not answer “should this be visible to a diagnosis program”.

The repaired version encodes each of those as an explicit rule:

from dataclasses import dataclass
from pathlib import Path

MAX_HITS = 20
MAX_SPAN_LINES = 120
MAX_FILE_BYTES = 256_000
EXCLUDED_DIRS = {".git", ".venv", "venv", "node_modules", "__pycache__"}
EXCLUDED_NAMES = {".env", ".env.local", "id_rsa"}

class ToolBoundaryError(RuntimeError):
    """Raised when a tool call would cross the declared boundary."""

@dataclass(frozen=True)
class SearchHit:
    path: str
    line: int
    text: str

class RepositoryTools:
    """Read-only repository access. The root is bound here, in application
    code, and is never exposed as a model-controlled argument."""

    def __init__(self, root: str) -> None:
        self._root = Path(root).resolve(strict=True)

    def _candidates(self):
        for path in self._root.rglob("*.py"):
            if EXCLUDED_DIRS.intersection(path.parts):
                continue
            if path.name in EXCLUDED_NAMES:
                continue
            if path.stat().st_size > MAX_FILE_BYTES:
                continue
            yield path

        def _resolve(self, relative: str) -> Path:
        target = (self._root / relative).resolve()
        if not target.is_relative_to(self._root):
            raise ToolBoundaryError(f"path escapes repository root: {relative}")

        rel = target.relative_to(self._root)
        if EXCLUDED_DIRS.intersection(rel.parts):
            raise ToolBoundaryError(
                f"path is inside an excluded directory: {relative}"
            )
        if target.name in EXCLUDED_NAMES:
            raise ToolBoundaryError(f"path is excluded: {relative}")
        if not target.is_file():
            raise ToolBoundaryError(f"path is not a regular file: {relative}")
        if target.stat().st_size > MAX_FILE_BYTES:
            raise ToolBoundaryError(f"file exceeds size limit: {relative}")

        return target

    def search_repository(self, query: str) -> list[SearchHit]:
        """Find lines matching a query. Returns at most 20 hits."""
        hits: list[SearchHit] = []
        for path in sorted(self._candidates()):
            text = path.read_text(encoding="utf-8", errors="replace")
            for line_no, line in enumerate(text.splitlines(), 1):
                if query.lower() in line.lower():
                    rel = str(path.relative_to(self._root))
                    hits.append(SearchHit(rel, line_no, line.strip()))
                    if len(hits) >= MAX_HITS:
                        return hits
        return hits

    def read_file(self, path: str, start_line: int = 1, end_line: int = 80) -> str:
        """Read a bounded span of a repository-relative file."""
        target = self._resolve(path)
        start = max(1, start_line)
        end = min(end_line, start + MAX_SPAN_LINES - 1)
        lines = target.read_text(encoding="utf-8", errors="replace").splitlines()
        return "\n".join(lines[start - 1 : end])

Three properties are worth stating explicitly, because they are the ones the rest of the chapter depends on.

The root is a constructor argument, so it is bound by the surrounding program. The tools the model can call take a query or a repository-relative path and nothing else. And sorted() in _candidates is not cosmetic: filesystem iteration order is not guaranteed, and an agent whose evidence changes between runs on an unchanged repository cannot be evaluated at all.

Mutation belongs behind a different boundary entirely. Nothing in this chapter applies a patch.

3. Attack the boundary you just built

Chapter 18 makes the general argument that a declared policy is not a firewall until something has tried to get through it. The same standard applies here, one chapter early, because a tool surface is the most concrete information boundary in the book.

The threat model has a specific shape. We are not defending against an adversarial model. We are defending against a program that will do whatever the tool signatures permit, and against a future edit that widens the surface without anyone noticing.

Here is the attack set the boundary should survive, and — this matters — the honest status of each against the measured run.

Four rows have measured evidence. One is an active path-escape probe; the other three are boundary or capability checks recorded by the run. Six further defences are present in code but unexercised. Chapter 18 later executes 288 constructed firewall attacks; this chapter’s evidence is deliberately narrower.

That distinction is the point. An unexercised defence is a hypothesis about your own code, and the book’s standard is that hypotheses get labelled as such until a run exists.

The declared-surface check deserves its own note, because it is the only one of these that catches a future mistake rather than a present one:

DECLARED_TOOLS = {"search_repository", "read_file", "inspect_symbol"}

def assert_surface(tools) -> None:
    actual = {t.__name__ for t in tools}
    if actual != DECLARED_TOOLS:
        raise ToolBoundaryError(f"undeclared tool surface: {actual ^ DECLARED_TOOLS}")

Six lines, and they are the difference between “the agent never called the dangerous tool” and “the dangerous tool was not available”. Only the second is evidence. The first is luck, and the run record cannot tell them apart after the fact.

4. ReAct as a bounded loop

DSPy documents dspy.ReAct(signature, tools, max_iters=...) as a generalized tool-using module over any signature, along with dspy.Tool wrappers and MCP conversion through Tool.from_mcp_tool(...). It is also mid-transition: dspy.ReActV2 is documented as experimental, native-tool-aware, and intended to become dspy.ReAct in a later release. Pin the DSPy version before depending on either name. This chapter’s runs are on 3.3.1.

The stable concept survives the rename:

signature
+
tool set
+
bounded reasoning/action loop
      ↓
structured output

The signature carries the contract, exactly as it has since Chapter 3. Note what is in it and what is not:

import dspy

class DiagnoseRepositoryIssue(dspy.Signature):
    """Diagnose a repository issue using only the supplied read-only tools.

    The repository root is application-bound and is intentionally not an input.
    Report the root cause, not only the proximate error.
    """

    issue: str = dspy.InputField()
    constraints: str = dspy.InputField()

    diagnosis: str = dspy.OutputField()
    suspected_files: list[str] = dspy.OutputField()
    missing_evidence: str = dspy.OutputField()

tools = RepositoryTools(root="/path/to/checkout")
assert_surface([tools.search_repository, tools.read_file, tools.inspect_symbol])

agent = dspy.ReAct(
    DiagnoseRepositoryIssue,
    tools=[tools.search_repository, tools.read_file, tools.inspect_symbol],
    max_iters=6,
)

missing_evidence is the most useful field here and the one most often left out. Without it the contract has no structured way to distinguish “I established a cause” from “I could not establish enough evidence.” A model can still write uncertainty inside diagnosis, but downstream software would then have to infer that state from prose. Giving the program an explicit place to put “I could not establish X” is a contract-level fix for a failure mode people usually try to solve with prompt engineering.

max_iters=6 is a budget, and budgets in this book are declared before the run and not adjusted afterwards.

5. The measured run

The fixture is a two-file package with one runtime defect. models.py defines Item.price. service.py sums item.cost. A focused execution reproduces the resulting AttributeError before the agent runs, so the failure is established independently of anything the model says about it.

The run is LM-backed on the canonical local model (ollama_chat/qwen3:latest, DSPy 3.3.1, temperature 0). The manifest records fixture, tool-boundary, provider and full-experiment fingerprints, and every tool observation is fingerprinted individually. The experiment lives in experiments/dspy-from-first-principles/ch14_react_tools — a directory name that predates the chapter renumbering and is worth correcting before the next run.

The agent used its full budget of six read-only calls:

1. search_repository(...)
2. search_repository(...)
3. inspect_symbol("total")
4. read_file("service.py")
5. inspect_symbol("Item")
6. read_file("models.py")

Read that trajectory as a reviewer would. It searches, narrows to the failing function, reads the file containing the call site, inspects the model class, reads the file defining it. It touches both implicated files and both implicated symbols. It does not loop, does not thrash, and does not stop early.

By every process criterion available, this is a good investigation.

Every safety and audit invariant passed. The diagnosis scored 0.60.

6. Right evidence, wrong answer

The agent concluded that Item has no cost attribute, and that this is what causes the runtime failure.

That is true. It is also the error message. The program spent six tool calls, 9,647 tokens and thirty seconds to arrive at a restatement of the traceback it started from.

The stronger diagnosis was available in its own observation stream. read_file("models.py") returned the definition of Item, including price. read_file("service.py") returned the call site using cost. Both facts were in context. The conclusion that joins them — service.py is using the wrong attribute name, and the fix is item.price — was never drawn.

This is the chapter’s result, and it is worth being precise about which failure it is not.

not an access failure     : the tools returned both relevant files
not a retrieval failure   : the agent found the right symbols unprompted
not a budget failure      : it had six calls and used them well
not a safety failure      : every boundary invariant held

a reasoning failure       : two facts in context, never joined

Every remedy that usually gets reached for here — better tools, a larger budget, richer search, more context — addresses a failure that did not occur. The agent’s problem was that it committed to a framing (“what attributes does Item have?”) and followed it to a locally consistent conclusion without reconsidering the alternative framing (“is the caller using the wrong name?”). That is a failure of reasoning path, and it is the exact gap Chapter 17 exists to attack.

Which produces the chapter’s rule:

A plausible investigation does not rescue a wrong conclusion.

The trajectory is evidence about process. It answers whether the program looked in the right places, stopped too early, looped, or used expensive tools it did not need. Those are real diagnostic questions and the trajectory is the only thing that answers them. What it cannot do is establish that the final answer is correct, and the temptation to let it try is strong precisely because a good trajectory is so much easier to inspect than a good conclusion.

7. What is 0.60 actually made of?

Chapter 9 established the book’s rule: a score is not a diagnosis. It applies to this chapter’s score too.

0.60 came from independent validation against known ground truth, which is the right structure — the agent did not grade itself. But it is a blended scalar, and a blended scalar over a diagnosis has the same defect the editorial metric had over a rewrite. It tells you the answer was partly right. It does not tell you which part, and therefore cannot tell an optimizer what to change.

The runner already decomposes the score. No post-hoc reconstruction is needed:

That is the exact arithmetic behind 0.60. The evaluator’s root-cause check sets root_cause_correct only when the combined diagnosis fields contain both cost and price. This run contained cost but never completed the connection to price, so it lost the entire 0.40 root-cause component while receiving every other component.

This also exposes a limitation in the evaluator itself. The metric does not contain a correct_repair component. We can observe from the stored diagnosis that the agent did not propose item.price, but that is a separate qualitative finding, not one of the terms that produced 0.60.

Chapter 14’s fix therefore applies in a more precise form. A scalar 0.60 can rank candidates; what it cannot do is tell reflective search what failed. Returning the scored components — for example root_cause_correct: false — creates a search-guidance channel without pretending the metric measured a repair it never scored.

As the repair program becomes capable of proposing patches, stronger externally checkable components become available:

8. One loop or an explicit graph?

A single ReAct loop fits work where the next action genuinely depends on what the last one returned. Our run is that shape: you cannot know to read models.py until the search has told you Item exists. The loop’s serial dependence is not overhead; it is the structure of the problem.

The loop pays for that flexibility twice. It re-reads its entire accumulated history on every step, so cost grows superlinearly in iterations. And it holds every dependency in context rather than in structure, which means a missed dependency is invisible — there is no place in the program where you could have declared it.

An explicit graph inverts both trades. Arachne is a useful real-system contrast because it treats the plan as a typed dependency graph: the dependencies live in the graph rather than in the model’s working memory, independent branches can run in parallel, and a missing edge is a structural defect you can inspect before executing anything. CodeSpy shows a different decomposition — parallel specialized reviewers followed by an auditor — which is the right shape when the subtasks are genuinely independent and the hard part is reconciliation.

The rule of thumb that falls out:

next action depends on last observation   → loop
dependencies known in advance             → graph
subtasks independent, merge is the work   → parallel + auditor

Sources: Strategic-Automation/arachne, khezen/codespy.

It is worth noticing that our failure is one a graph would not have fixed. The agent had both facts. No amount of dependency structure joins them for it. That is a genuine limitation of this chapter’s remedy set, and it is why the next two chapters exist.

9. What this run establishes, and what it does not

One fixture, one defect, one session, one score. The book’s own standard — a 44-case corpus, seven paired sessions, a stated noise floor — is nowhere near met here, and the result should be read accordingly.

The safety result is the strongest thing here, and it is still narrow. Four measured boundary checks passed; one of them was an active path-escape probe. Six further defences are written and unexercised. The declared-surface assertion has never had to fire.

The diagnosis result is n=1. A single run scoring 0.60 on a single fixture establishes that this failure mode occurs. It does not establish a rate, and nothing here supports a claim about how often agents stop at the proximate cause.

The fixture is small enough to be unrepresentative in a specific direction. Two files makes evidence selection unusually easy. A larger repository introduces more opportunities to miss, bury, or retrieve the wrong evidence, but this n=1 run does not establish how diagnosis accuracy changes with repository size. Do not read 0.60 as a floor, a worst case, or a rate.

The score decomposition is measured, while the repair diagnosis is partly interpretive. Section 7’s four weighted components are the actual runner metric. The additional observation that the agent failed to propose item.price comes from reading the stored answer against fixture ground truth; correct_repair was not a scored component.

What survives all of that is the structural claim, and it does not depend on the numbers:

Safety and correctness are independent properties of an agent, and a run can pass every boundary check while producing an answer that would break production.

That separation is the reason this chapter measures the two things apart. A tool-boundary audit that reported a single combined score would have been unable to say it.

What Usually Goes Wrong

Conclusion

We gained environment access without giving up the program abstraction. An agent is a DSPy program with a signature, a bounded tool surface, an iteration budget, a trajectory and an evaluation protocol. Everything the book established about contracts, metrics, splits and promotion continues to apply.

The measured run separated safety from quality, and they separated cleanly.

The boundary held. The root was application-bound and absent from the signature. A path-escape probe was blocked. A sentinel outside the repository never appeared in any observation. No mutation, shell or network capability was exposed. The four measured boundary checks passed — including the active escape probe — and six further defences remain written but unexercised, which makes them hypotheses about our own code rather than results.

The agent then produced a six-call trajectory that touched exactly the right files and symbols, held both halves of the defect in context simultaneously, and concluded by restating the error message it started from. It scored 0.60: the runner awarded 0.25 for identifying the suspected file, 0.20 for inspecting relevant evidence, 0.15 for valid tool use, and 0.00 on the 0.40 root-cause check. The correct repair was also absent, but the runner did not score repair correctness as a separate component.

Safe tools can deliver correct evidence to reasoning that still arrives at the wrong answer.

We removed the assumption that tool use sits outside the optimization story. We removed the assumption that a plausible trajectory justifies its conclusion. And we established that a blended diagnosis score is as unhelpful to an optimizer here as the blended editorial score was in Chapter 9 — the same defect, in a new object.

Two problems remain, and they are different from each other.

This fixture had two files. A real repository has thousands, and a bounded tool surface does not tell the program which of them to look at. That is a selection problem, and it is the next chapter.

Then there is the failure we actually measured: two facts in context, never joined. No tool and no retriever fixes that. It requires the program to consider more than one reasoning path before committing to a conclusion, which is Chapter 17.

Further Reading

• ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., 2022): the interleaved reason-act-observe loop dspy.ReAct implements. (arXiv:2210.03629)
• Toolformer (Schick et al., 2023): an earlier framing in which tool use is learned rather than orchestrated. (arXiv:2302.04761)
• SWE-bench (Jimenez et al., 2023): repository-level repair as a benchmark, and the source of most current intuitions about how hard this task is. (arXiv:2310.06770)
• SWE-agent (Yang et al., 2024): argues that the agent-computer interface — the tool surface itself — is a primary determinant of performance. (arXiv:2405.15793)