Preserve Before You Interpret

Part 3 — Give Intelligence a Runtime

Raw output first

Every rule an AI application applies to a model’s response is an interpretation. Did generation finish? Is this valid JSON? Is this a refusal? How many tokens did it cost? What does the answer actually say? Those rules can turn out to be wrong: a provider adds a field, a parser misses a case, a failure shows up that nobody anticipated.

When that happens, you want to apply the corrected rule to the evidence you actually preserved. If all you kept was extracted text and a status, you can reconsider only what those fields retained; you cannot recover a discarded finish reason, error body or other response detail. Asking the model again does not restore the old observation. It creates a new request and, for a stochastic call, potentially a different generation.

observation  ≠  interpretation  ≠  decision

Preserve the observation once. Interpret it as many times as you need, with each reading versioned and none of them rewriting the others. By the end of this chapter you will be able to design model calls so that yesterday’s conclusions can be revised from yesterday’s bytes, without a new request and without editing history, and so that when the bytes are gone, the system refuses to pretend otherwise.

The success that was cut off

Chapter 11’s fourth live run was recorded as a success. The provider had answered with HTTP 200 and text, and CodeAI’s interpreter at the time treated any answer with text as complete. Opening the preserved response told a different story: "finish_reason": "length". The model had used every output token it was allowed, and the review stopped mid-sentence.

That mistake was found by a person, reading bytes that happened to have been kept. Chapter 12 then taught the adapters to read the finish reason. Chapter 13 built better readings as pure projections over preserved responses, and stopped deliberately short of letting them change anything: past decisions stayed bound to the interpretation in force when they were made.

So the record still said “succeeded”. A better interpreter existed, and nothing in the runtime could act on it.

Why must the underlying observation outlive the interpretation?

Observe today, interpret tomorrow

The honest sequence in a system that calls models looks like this. Observe today. Interpret today. Discover tomorrow that the interpreter was wrong. Then interpret again, tomorrow, without repeating yesterday’s request.

Yesterday’s request is not repeatable in the sense this chapter needs. It was a stochastic external effect that may also have consumed billable or quota-limited provider capacity. Asking again creates another request and may produce a different generation, so it cannot answer the historical question “what did we actually receive on the day we decided?” Chapter 16 established the recovery consequence: once a provider may have served the request, this process cannot roll that effect back.

Chapter 16 ended with one state its work-state projection could name but not act on: observed, reinterpret. The response had arrived and been stored, and the process died before interpreting it. resume_call refused, correctly, because nothing in CodeAI yet ran an interpreter over stored bytes.

This chapter closes that state, and it rests on a distinction:

• A restart can tell us that the response has already arrived. It cannot reinterpret a response we failed to keep.
• Resumption requires facts. Interpretation is not one of them.

Facts, and what they are taken to mean

Pat Helland’s Immutability Changes Everything separates two kinds of entry in a ledger: observed facts, which are recorded and kept, and derived facts, which are calculated from them when needed (Helland, 2015).

His example is accounting. A received payment is observed; a balance is derived. Accountants don’t use erasers: a correction is a new entry, never an edit to an old one. Of a database’s transaction log he writes, “The truth is the log.” The database is a cache of a subset of it. He is equally blunt about the limit: having the right bits is not enough, because you still have to know how to interpret them.

Quinlan and Dorward’s Venti archival server gives the storage half of the argument (Quinlan and Dorward, 2002). Each block is addressed by a hash of its contents, so a block cannot be modified without changing its address, and storage is write-once by construction. The authors argue that a write-once policy greatly reduces the opportunities for data loss.

The AI evaluation literature has made the same choice for a different reason. HELM released all of its raw model prompts and completions publicly, for further analysis (Liang et al., 2023).

None of these papers is about model runtimes; the mapping is this chapter’s. For a CodeAI call it comes out as four layers:

Only the first row is the preserved observation. Even that is evidence about what CodeAI’s transport path recorded, not proof of the remote provider’s internal state or of the truth of the content. Everything below it is a conclusion, and conclusions are allowed to be wrong provided the observation they were derived from is still available.

This is not a normalization chapter. Chapter 13 asked what usage numbers mean. This one asks what has to be true for any meaning to be revised later, after decisions have been built on it.

How CodeAI reinterprets

The difference between the two completion readings fits in a few lines. For generation completion, interpreter v2 maps explicit provider completion reasons per API dialect and leaves an unrecognized reason unknown rather than inferring completion from prose. The full interpreter also classifies transport and error evidence; the excerpt below isolates only the completion rule. Reduced from CodeAI’s interpretation module:

_COMPLETION_REASONS = {                          # excerpt
    ("chat_completions", "stop"):              "complete",
    ("chat_completions", "length"):            "truncated",
    ("messages", "max_tokens"):                "truncated",
    ("responses", "max_output_tokens"):        "truncated",
    ...
}

# The two rules, reduced from interpret_v1 and interpret_v2:
v1_state = "complete" if output_text else "empty"                          # text means complete
v2_state = _COMPLETION_REASONS.get((protocol, finish_reason), "unknown")   # the provider's reason decides

A response with text and finish_reason: "length" is complete under the first rule and truncated under the second. Nothing about the bytes differs; only the reading does.

CodeAI makes the second reading actionable after the fact with two operations. The day-two process in the experiment below made exactly this call:

runtime.reinterpret_call(CALL, interpreter_version=INTERPRETER_V2,
                         policy_version=ATTEMPT_POLICY_V2)

reinterpret_attempt reads the attempt.observed event and the bytes it names, and nothing else: no derived copy, no adapter-authored messages. It verifies the bytes against their recorded hash, applies the named interpreter, and appends one attempt.interpreted record, marked as recorded by reinterpretation and causally linked to the observation. It needs no completion record, so it works on an attempt whose process died after the response was stored, and it is idempotent per version.

reinterpret_call interprets every attempt and derives every decision before appending anything. Only then does it append any missing interpretations and one call.reinterpreted record, holding the new status and reason, the per-attempt decisions, the interpretation IDs and observation hashes it used, the prior status and the record that held it, and provider_effect: false. It never edits call.status_decided or starts an attempt, staying idempotent per version pair.

The work-state projection and acceptance validation now adopt the latest of call.status_decided and call.reinterpreted, and the projection says which one it used (status_basis). Neither operation ever contacts a provider. If the bytes cannot be read, both refuse without appending anything.

One preservation, many readings over time:

    flowchart TD
    B(["raw response bytes<br/><i>preserved once</i>"]) --> H["content hash<br/><i>verified before every reading</i>"]
    H --> A["immutable artifact<br/><i>never rewritten</i>"]
    A --> V1["parser v1<br/><i>text means complete</i>"]
    A --> V2["parser v2<br/><i>the provider's reason decides</i>"]
    V1 --> I1["interpretation: complete"]
    V2 --> I2["interpretation: truncated"]
    I1 -.->|"same bytes, new reading"| I2
    I2 --> R["call.reinterpreted<br/><i>appended; history never edited</i>"]
  

Recorded, reinterpreted, refused

The demonstration was preregistered before it ran. Each phase ran as a separate operating-system process, standing in for work done on different days. A synthetic provider appended an fsync’d line to its own receipt log for every request, so provider effects are counted outside the ledger, and outbound connections were refused throughout.

The response is a synthetic fixture. It reproduces Chapter 11’s defect class rather than its bytes: text, with finish_reason: "length". Interpreter v1 reproduces the historical rule (text means complete). Interpreter v2 lets the finish reason decide.

Day one, day two, day three

Day one. One provider request, one receipt, ten events. Interpreter v1 read the response as complete, the attempt decision recorded “v1: no error recorded”, call.status_decided recorded succeeded (“final attempt accepted”), and the work state projected the task’s next step as “check and accept”.

A counterfactual v2 projection, run in the same inspection, already said unresolved, and recorded nothing. The better answer was computable on day one, but it had no standing.

Day two. A different process opened the same files and called reinterpret_call under v2.

It appended two events. The first was a v2 interpretation, truncated for provider reason length, caused by the original attempt.observed event and naming body ff29c451…. The second was call.reinterpreted, recording unresolved (“generation truncated, not treated as completed cognition”) with prior status succeeded.

The provider’s receipt log still held one line, and the response file still hashed to the value observed on day one, with the ten day-one events still first and unchanged. The v1 interpretation and the day-one decision are still there; they are no longer adopted.

Repeating the reinterpretation appended nothing and returned the same record.

Day three. A third process projected the work state. The call is unresolved, on the basis reinterpretation:attempt-interpretation-v2/attempt-policy-v2, and the task’s next step is no longer “check and accept” but “start the call”.

Then someone tried to accept the task anyway, citing the day-one v1 interpretation, with a check on the output text that passed. Acceptance was refused for two reasons: source_call_not_succeeded and interpretation_not_decision_basis. The first holds because the adopted status is now unresolved. The second holds because the cited interpretation is no longer the one the adopted status rests on. The ledger ended at fifteen events.

Nothing was asked of the model after day one; what changed was what the system concluded from what it already had, and therefore what it would do next.

The attempt killed after the bytes arrived

The producer was killed with Popen.kill immediately after attempt.observed was committed. The ledger held five events and the receipt log one line.

A new process found the stage “observed”, with next operation “reinterpret”. The old projection, interpret_attempt_as, returned nothing, because it needs a completion record that was never written.

reinterpret_attempt read the preserved bytes and recorded truncated: one event appended, receipts still one. The work state moved from observed, reinterpret to interpreted, redecide.

That is the first post-effect state from Chapter 16’s table for which CodeAI now has an explicit recovery operation over preserved evidence. It does not happen automatically inside resume_call: this run invoked reinterpret_attempt deliberately. Redeciding and finalizing are still not automated.

A quiet substitution

Building reinterpretation started with a question about CodeAI’s existing projection, interpret_attempt_as: what did it do when an observation’s bytes were gone? It did not refuse.

If the response file was missing or failed its hash check, it quietly fell back to a copy CodeAI had derived from the response at execution time. It then attached the observation’s artifact reference to the result anyway. A reader would see an interpretation naming bytes that did not exist.

That was fixed on its own, before the feature. A missing or corrupt body now raises ObservationUnavailable. The regression test fails against the code before the fix.

The derived copy is exactly the kind of thing this chapter warns about. It was an interpretation, and it was being presented as the observation.

Without the bytes

Two copies of the day-one directory were damaged. In one the response file was deleted; in the other its bytes were altered and the recorded hash left alone.

Both the projection and the reinterpretation raised ObservationUnavailable. One said “response body bytes are missing”; the other said they “do not match their recorded sha256”. The ledgers stayed at ten events, and the receipt logs at one.

The runtime can still tell you that a response arrived, when, and what hash it had. It can no longer apply a body-dependent rule written later because the body that rule would read is gone. From CodeAI’s own preserved record, that historical question is now unanswerable. Asking the provider again would create another request and another response, not recover the missing observation.

The fourth copy, also with its body deleted, was read by CodeAI as it stood before the defect fix, checked out separately. It returned an interpretation, truncated for reason length, and named artifact ff29c451… as its evidence. The file with that hash was not on disk.

The conclusion happened to be right, because the derived copy it silently used still held the finish reason. That is the dangerous version of this failure. A wrong answer gets investigated. A right answer that names evidence which is not there passes review, and it stays wrong about its provenance for as long as anyone relies on it.

What this is not

• Not a proof that v2 is right. It is right about this defect class, on this fixture. v3 over the same bytes is the expected future, not a failure.
• Not undo. Anything done under the day-one conclusion stays done. A task accepted, a retry not taken, or an email sent under v1 is still in history. This run did not exercise a task that was already completed before its source call was reinterpreted.
• Not evidence of truth. A matching hash proves the bytes are the ones that were stored. It does not prove they are what the provider sent, or that what the model said is correct.
• Not normalization. The interpreter changed; the point is when it can change, and what it may and may not touch when it does.

Where it is still weak

1. Only preserved observations are reinterpreted. Adapter-authored error text is not an observation and is not used, so an attempt whose original interpretation relied on it may not replay identically.
2. Attempts without attempt.observed cannot be reinterpreted at all. That includes fake adapters and older records.
3. Latest record wins. There is no precedence between versions; reinterpreting with an older version afterwards would be adopted.
4. Redecide and finalize are not automated for interrupted attempts.
5. The derived copy still exists for attempts whose observation names no body.
6. A synthetic provider, one defect class, one fixture, a single writer.

Do this now

Thirty minutes. Find out whether you could change your mind about yesterday.

1. Pick one model call your system made yesterday. Find what it stored. Is it the provider’s response bytes, or only your extracted text, status and token counts?
2. Write down one rule your code applies to responses: success detection, JSON extraction, refusal detection, usage accounting. Suppose it was wrong. Could you re-run a corrected rule over yesterday’s calls without calling the model again?
3. Find where the result of that rule is stored. When you fix the rule, does the old result get overwritten, or does a new, versioned result sit beside it?
4. Find one downstream thing that trusted the old result: a status, a retry that didn’t happen, a merged change. Would it notice?

If you are building with an assistant:

Make model-call interpretations revisable without repeating the call.
- Preserve the transport observation before the runtime records an
  interpretation or decision. When a response body exists, store its exact
  bytes content-addressed and have `attempt.observed` name the bytes and hash.
  Adapter parsing may already have occurred; later reinterpretation must read
  the preserved observation rather than a derived copy.
- Record every interpretation with an interpreter version, linked to the
  observation it read. Record decisions with a policy version.
- Add reinterpret(call, interpreter_version, policy_version): verify the
  bytes' hash, compute every new interpretation and decision first, then
  append them plus one "call reinterpreted" record naming the prior status.
  Never edit earlier records. Idempotent per version pair. Never call the
  provider.
- If the bytes are missing or fail their hash, refuse and append nothing.
  Never fall back to a derived copy while naming the original bytes.
- Make status projections and acceptance use the latest adopted record.
- Test across separate processes: record under the old rule, reinterpret
  under the new one, and show provider request count unchanged, bytes
  unchanged, old records intact, next step changed. Delete the bytes on a
  copy and show that reinterpretation is refused.

Failure modes

• Storing only the interpretation. When the rule turns out to be wrong, there is nothing to apply the corrected rule to.
• Asking the model again. It creates a new request, may consume billable or quota-limited provider capacity, and cannot recover the response that existed when the original decision was made.
• Overwriting the old conclusion. You lose what decisions were actually based on.
• A better interpretation with no standing. Chapter 13’s projections were correct, and nothing acted on them.
• Substituting a derived copy for missing bytes. The conclusion may even be right. Its provenance is false.
• Trusting a hash as truth. It proves the bytes are unchanged, not that the content is correct.
• Assuming reinterpretation undoes effects. It changes what the system concludes next, not what it already did.

What this chapter established

• Observation ≠ interpretation ≠ decision. The transport observation is preserved evidence: metadata plus exact response-body bytes when a body exists. What that evidence means and what to do about it are conclusions. A matching hash establishes integrity of the stored bytes, not the provider’s internal state or the truth of the content.
• Preserve once, interpret many times. Persist the transport observation before the runtime commits interpretation or decision records. Record every interpretation with a version, and every decision with a policy version. Helland’s observed and derived facts, Venti’s write-once content addressing and HELM’s release of raw completions support the reasoning; the mapping onto model runtimes is the book’s.
• Reinterpret; do not re-ask. A corrected rule applied to preserved evidence answers what the original call left in the record. Asking the model again creates a new call and may produce a different response.
• Append; never edit. An attempt-level reinterpretation appends another attempt.interpreted record. A call-level call.reinterpreted record is what changes the adopted call status. Earlier interpretations, decisions and effects stay in history.
• Do not substitute for missing evidence. When an observation references response-body bytes and those bytes are missing or corrupt, body-based reinterpretation must refuse. A no-response observation can still carry transport metadata; a derived copy must not be presented as the missing body.

What CodeAI showed. CodeAI now refuses, instead of silently substituting a derived copy, when an observation references response-body bytes that are missing or corrupt. reinterpret_attempt derives a new interpretation from the preserved observation; reinterpret_call does that for every recorded attempt, re-applies the named decision policy, and appends a versioned call-level status without contacting the provider or editing history. Work-state and acceptance then adopt the latest call-status record.

In separate processes, a truncated response recorded as a success on day one was reinterpreted on day two as unresolved, with no new provider request and with the bytes and day-one history unchanged. The task’s next step changed from “check and accept” to “start the call”, and an acceptance citing the old interpretation was refused. An attempt killed after its response was stored was advanced from the bytes alone. With the bytes deleted or corrupted, reinterpretation was refused and nothing was appended, while the code from before the fix returned an interpretation naming bytes that were not on disk. Not established: that v2 is correct in general, undoing effects made under an old conclusion, version precedence, or automated redecide and finalize.

Evidence notes

Independent verification. The bundle’s verifier imports neither CodeAI nor the producer. It reads the preserved bytes, applies its own statement of both rules — under v1 text means complete, under v2 the finish reason decides — and checks the recorded conclusions against that reading: day one’s success, day two’s truncated and unresolved, and the counterfactual. It also requires the v2 interpretation to be caused by the observation and to name the same bytes, the receipt count not to move during reinterpretation, and day one’s events to be an unchanged prefix of day three’s, and it checks the killed attempt, both refusals, the before-fix probe, the preregistration order, and that the run used committed code.

All 17 semantic claims pass; the full run, adding byte hashes for 414 files, exits 0. A fresh CodeAI process reopened the ledgers and projected exactly the work states the original inspections recorded.

Five seeded corruptions were run with the byte inventory bypassed, so only meaning could catch them:

Next

An interpretation is still one conclusion about one response, held by the runtime. A review that says a claim is unsupported is a response, too. So is the justification for accepting a task.

The next step is to break model-derived conclusions into claims that name their evidence, then bind decisions to the claim standing they actually relied on. Chapter 18 carries the adopted source-call status into that standing, so a later reinterpretation can change the basis without rewriting the original decision and the runtime can name which decisions now rest on moved evidence.

Continue with Claims, Evidence, and Decisions.

References

• Pat Helland. Immutability Changes Everything. 7th Biennial Conference on Innovative Data Systems Research (CIDR), 2015. https://www.cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
• Percy Liang, Rishi Bommasani, Tony Lee, et al. Holistic Evaluation of Language Models. Transactions on Machine Learning Research, 2023. arXiv:2211.09110. https://arxiv.org/abs/2211.09110
• Sean Quinlan and Sean Dorward. Venti: A New Approach to Archival Storage. Proceedings of the FAST 2002 Conference on File and Storage Technologies, USENIX, 2002. https://www.usenix.org/legacy/publications/library/proceedings/fast02/quinlan/quinlan.pdf

Implementation sources:

• The defect fix: src/codeai/interpretation.py (ObservationUnavailable); src/codeai/runtime.py (interpret_attempt_as refuses instead of substituting); regression test in tests/test_interpretation.py.
• Reinterpretation:
  • src/codeai/runtime.py: reinterpret_attempt, reinterpret_call, _interpret_from_observation, _append_reinterpretation, _latest_status_record.
  • src/codeai/providers.py: output_text_for.
  • src/codeai/interpretation.py: _COMPLETION_REASONS, interpret_v1, interpret_v2 (the reduced rules and map excerpt above; inspected in current source).
  • src/codeai/workstate.py: adopted status with status_basis.
  • src/codeai/acceptance.py: adopted status in _validate_source.
  • Tests: tests/test_raw_output.py (9); full suite 327 passed.
• Evidence: experiments/applied-ai/evidence/raw-output/2026-09-13-a451e34/.
  • The preregistration and execution record.
  • Five cases, each with an SQLite ledger, artifact store, provider receipt log, per-day inspections and exported events.
  • The independent verify.py, five seeded corruptions, test outputs, chapter-evidence-report.md and hashes.json.
• Producer: experiments/applied-ai/raw_output_demo.py. The executed copy is pinned as the bundle’s run.py.