Debugging by Layer

The extension worked yesterday. Today the same command does nothing, the transcript looks fine, and there is no error anywhere. You have no stack trace. You have a model that did something, a process that recorded it, and several places it could have gone wrong.

Debugging an agent badly means looking everywhere at once. Debugging it well starts from the architecture you now have. The layers introduced in chapter 1, refined by the tools, extensions, sessions and interfaces covered since then, give you a list of suspects. Controlled substitutions can narrow that list, provided you track what each substitution changes.

The suspects

interface              terminal, print, JSON, RPC, SDK: how you watch it
host / application     extensions, settings, trust, context files, sessions
tool / effect          what a tool did, and what a gate let through
agent core             the loop: turns, hooks, events, queues
model access (pi-ai)   one request: auth, payload, stop reason, usage
provider               the service that answered

Start with the cheapest layer to check, not a rule about which layer is usually at fault. For “the model ignores my instruction”, the first thing to suspect is something upstream of the request — a prompt section that did not load, a context file that was not admitted, an entry that was compacted away — because those are free to inspect. That is a starting heuristic and not a diagnosis: a provider that silently drops a parameter looks identical from the outside, and ruling it out needs the same replacement test as everything else. “The agent keeps retrying” is the provider and the retry layer. “My command does nothing under --print” is the interface. The work is in assigning the symptom to a layer by experiment, not by expectation.

The one question that works

Change one layer or dependency at a time. Does the symptom survive under otherwise equivalent conditions?

A scripted provider and Pi’s separable components make some of these checks practical. Each substitution is diagnostic evidence, not proof that a particular layer is defective:

This is chapter 1’s replaceability test used as a diagnostic. Record the request, tools, configuration, working directory and observed result before each comparison. If a substitution changes more than one of them, it may still narrow the investigation, but the result cannot isolate a single cause.

    flowchart TD
  S["Record the symptom and a repeatable case"] --> Q1{"Reproduces with a scripted provider?"}
  Q1 -->|"yes"| A["Inspect host, loop, tools and extension paths"]
  Q1 -->|"no"| B["Compare live-model output, provider payload and changed inputs"]
  A --> Q2{"Reproduces with extensions disabled?"}
  Q2 -->|"no"| EXT["Re-enable extensions one at a time"]
  Q2 -->|"yes"| Q3{"Reproduces with an equivalent bare Agent?"}
  Q3 -->|"no"| HOST["Inspect coding-agent integration and missing features"]
  Q3 -->|"yes"| CORE["Inspect the shared loop and tool implementation"]
  B --> CHECK["Check authentication, adapter and provider response"]
  EXT --> VERIFY["Confirm with an isolated reproduction"]
  HOST --> VERIFY
  CORE --> VERIFY
  CHECK --> VERIFY
  

Layer by layer

For each layer: what its failures look like, and the instrument that answers the question in one step.

Provider

Looks like: a 529 and a pause, a request that hangs, an answer that is cut off, a rate limit.

Instruments: retry and recovery events (auto_retry_start, compaction_end with reason: "overflow") reveal how Pi classified and handled an error; they do not by themselves establish its root cause. Chapter 29 is the reference. provider_stream_event exposes parsed provider events before Pi normalises them. Treat them as read-only, since mutation can affect normalisation, and do not mistake them for original HTTP bytes. The shipped debug-provider.ts example groups these events by assistant message. If you suspect the provider adapter, compare the parsed events with the eventual assistant message.

Model access (pi-ai)

Looks like: a model that loads but does not appear in /model, a call that returns stopReason: "error", a credential that works in your shell and not in Pi.

Instruments: check the credential sources and their precedence from chapter 3, then inspect stopReason and errorMessage when a request returns an assistant error result. Do not assume every failure is represented by a result: setup and invocation errors can also throw. pi-ai’s onPayload callback lets you inspect the provider payload prepared by its adapter before transmission. It is not a packet capture, and the payload may contain sensitive prompts or tool arguments:

const response = await models.complete(model, context, {
	onPayload: (payload) => console.log(JSON.stringify(payload, null, 2)),
});

Documented, not run: this onPayload pattern comes from the pi-ai README, which documents it for stream, complete, streamSimple and completeSimple. Use it to inspect the outgoing payload when a provider rejects a request or a tool declaration seems to be missing. Redact credentials, prompts and private data before saving or sharing the captured payload.

Agent core

Looks like: the loop does one turn too many or too few, a tool is never called, a hook seems not to fire, a steering message arrives late.

Instruments: the event stream and the observed hook order from chapter 39. Subscribe and print event.type. If agent_end fired and work continued, you are looking at the coding agent’s retry, recovery or queue (chapter 25), because the core’s run really is over. Recall that a finishTurn that returns continue unconditionally is an infinite loop. If a guard sees stale state, recall that raw agentLoop() does not wait for your handlers and the Agent class does.

Tool and effect

Looks like: the model “keeps retrying”, a gate that is “too strict”, a write that happened when it should not have.

Instruments: a tool_call handler that fails blocks the tool as a fail-safe, and the model sees a failed result, not a crash, so an over-strict gate looks like a model that will not take no for an answer. A tool that returns an object saying “failed” has not failed: only throwing, or isError: true, marks it. For an MCP tool, a gate matching only built-in names will skip the mcp__<server>__<tool> form (chapter 21); match the actual registered name. And remember the matrix from chapter 39: if the effect has already happened, no later hook explains why. The question is which hook was supposed to refuse it before.

Host and application

Looks like: an extension that “silently did nothing”, a setting that does not take, a resumed session that is the wrong one, instructions that are not followed.

Instruments: four documented lifecycle rules explain most of these.

1. The factory is not a place to start long-lived processes. “Do not start processes, sockets, watchers, or timers in the factory because some invocations load extensions without starting a session.” A factory may also run in print mode; work started there is not automatically suppressed by --print. Register in the factory, and start session-scoped resources from session_start or the operation that needs them.
2. Command-context operations are command-only. Waiting for idle, reloading and session replacement are not on ExtensionContext because calling them from lifecycle handlers can deadlock the runtime. A handler that hangs: check this first.
3. Reload replaces the runtime. State captured before await ctx.reload() must not be used after it.
4. A handler failure is not a no-op and is not a crash. Pi reports handler errors and continues where possible.

For state rather than lifecycle, /session shows the session file, ID, message count, token usage and cost. If you resumed the wrong session, check the working directory, selected session path and active branch. The model does not receive the entire session file; compaction and context transformations can further change the active-branch projection. When a long session degrades, inspect the compaction outcome and, if the question is what the model actually received, capture the provider request rather than inferring it from the stored history.

Be precise about export: pi --export <session.jsonl> [output.html] generates an HTML export from a session file; interactive /export writes an export artefact, with formats depending on the selected option. Neither command is equivalent to copying the original JSONL file. For a raw append-only record, inspect or make a protected copy of the session’s .jsonl file. For “what did the model receive”, neither raw JSONL nor a rendered export is sufficient: transformContext, convertToLlm and prompt sections can alter the request. Chapter 27 explains the stored record, and chapter 18’s tests illustrate request-level observation.

Interface

Looks like: a key that does nothing, a rendering glitch, a command that works interactively and not in a script.

Instruments: check the mode first. A ctx.ui.confirm() in a --print run is not a bug, it is a documented absence of UI, and the ctx.hasUI and ctx.mode === "tui" guards are the fix. For rendering, PI_TUI_WRITE_LOG captures the raw ANSI stream, and /hotkeys is the authority on which shortcuts are live. Appendix B has the terminal detail.

The artefacts

Some tools cut across layers. They are worth knowing by name, and the first one comes with a warning.

Review pi-debug.log before sharing it. It can contain prompts, model responses, tool output, file contents and terminal data. The same warning applies to /export, /share and /bug. “Without credential values” describes the report format, not a promise about its contents: it will still contain your conversation if the diagnostics include it, which is why /bug lets you omit the transcript or have the current model summarise the problem instead.

When the layers disagree

The hardest failures are the ones where two layers each look correct. Two patterns cover most of them.

A symptom that changes when you replace a layer. You swap the live provider for the faux provider and the bug vanishes. That shows the symptom is not reproduced by your scripted responses; it does not prove the model is at fault. The scripted tests in chapter 41 test specified behaviour around the model, not the real model’s reliability. Compare request inputs and repeat the real-provider trial under controlled conditions before assigning the cause: a failure that occurs one run in five can easily be missed in a single run.

An effect with no cause in the transcript. Something changed on disk and the transcript does not show a tool call that did it. Look at what else runs with the process’s permissions: an extension’s own code, a package’s postinstall, an MCP server you launched. This is chapter 42’s subject: the transcript is a record of what the model asked for, not of everything the process did.

What you can and cannot claim

Documented: the artefacts (usage.md, sessions.md, tui.md), the extension lifecycle rules (extensions.md), onPayload and onProviderStreamEvent (pi-ai README), and the loop and hook behaviour cited from chapters 29 and 39.

Proposed: the layer list, controlled-substitution method and decision diagram. The method is this book’s formulation, informed by the separations discussed in chapters 1 and 37. It has not been validated against a representative set of real failures; treat it as an investigation heuristic, not a documented Pi procedure or a proven fault-localisation algorithm.

Next

You now have a way to narrow down which layer a failure may belong to. That closes Part III. The next part changes the subject from where a failure lives to how the same agent is driven, and the first question about any interface is how much of the agent it can see.

Chapter 31 asks whether the interface is part of the agent at all.