Evaluation on AIBussin — AI applications, systems and books

The Measurement Instrument

Mon, 14 Sep 2026 05:00:02 +0100

Chapter 1 ended with a test. Does the past change what the system does now, and does it change it for the better?

That is easy to state and hard to run. This chapter builds the thing that runs it.

Two words will recur for the rest of the book, and they must not blur together. The memory system is the thing that remembers. The instrument is the apparatus that decides whether it does. They are separate products, and this book builds both.

Meat Proxy

Mon, 14 Sep 2026 05:00:05 +0100

Part 1 — Where You Stand

Review degrades because the system succeeds

Ten weeks

Week one, the model drafts a paragraph review and you read every line of it. You disagree with two of its four flags, check a source yourself, rewrite one sentence. The whole thing takes twenty minutes and you feel you have earned the output.

Week three, the reviews have been good. You read them properly but you stop re-checking the sources it says it checked. Nothing bad happens.

The Memory Nexus

Sat, 19 Sep 2026 05:00:06 +0100

Chapters 3 through 5 leave the system with an embarrassment of options. Chapter 3 built hybrid retrieval over raw history with reranking. Chapter 4 added a persistent derived graph with several query modes. Chapter 5 added cue-conditioned associative propagation over that graph. Each chapter earned its mechanism conditionally, and each left the cheaper layers available underneath. The question none of them answers is the one a deployed system meets first:

Intelligence in the Wrong Direction

Mon, 14 Sep 2026 05:00:07 +0100

Part 1 — Where You Stand

Capability without an objective is magnitude without direction

The thing the marketing does not cover

Every vendor will sell you more capability. None of them can tell you whether that capability moves your process toward its objective without a measurement you supply.

A stronger model can execute the wrong objective more effectively just as it can execute the right one more effectively. Capability alone does not tell you whether the gap between what you wanted and what you got narrowed or widened. That is an empirical question.

Examples Are Experimental Data

Fri, 28 Aug 2026 10:30:00 +0100

Chapter 6 gave the program a dependency boundary and, more importantly, evidence that identical recorded program state can still produce different outputs. We can now say what the program is, what ran it, and why a small score delta cannot be interpreted without repeated measurement.

What we still cannot do is say whether the program is any good, because we have nothing to measure it against.

Examples in DSPy are not decoration around the program. They are experimental data with several possible roles. Optimization consumes that evidence: training data may shape candidate state, development data may select among candidates, and holdout data must remain outside both processes if it is to support an independent claim.

You Cannot Optimize What You Cannot Measure

Fri, 28 Aug 2026 10:35:00 +0100

Chapter 7 produced material. This chapter produces evidence.

That makes optimization possible. It does not yet make optimization safe.

no metric
 → no systematic optimization

metric
 → behavior can be optimized

wrong metric
 → the wrong behavior can be optimized systematically

This chapter establishes the surface on which search can operate. Chapter 9 asks what that surface rewards when the metric becomes the target.

 flowchart TD
 P[frozen program] --> E[evaluate]
 C[frozen cases] --> E
 M[metric] --> E
 E --> PC[per-case results]
 PC --> AG[aggregate result]
 PC --> FI[failure inspection]

No optimization happens here. Note that the per-case results feed two readings — the aggregate and the failure inspection — and this chapter’s finding comes from the second. The question is narrow:

What Survived the Transformation?

Fri, 25 Sep 2026 02:45:00 +0100

Part I has spent five chapters transforming information. Visual previews compress pages into triage cards. The router adds diagrams, tables, and timelines — or declines. The semantic record stabilises structure. The sidecar brings in surrounding material. Relation readout classifies how that material bears on the source. Every one of these operations can fail silently: the output looks fluent, looks professional, looks right, while the content has shifted underneath. A diagram drops the exception. A summary upgrades a correlation to a cause. A sidecar panel restates the source’s speculative conclusion as its headline finding. Nothing in the rendering announces the damage.

When the Metric Becomes the Target

Fri, 28 Aug 2026 10:40:00 +0100

Chapter 8 built a metric and found a defect in it by reading a table. Six cases where the correct answer is capped at 0.65, discovered by grouping per-case scores and noticing that a whole category sat on an exact number.

That was luck dressed as diligence. This chapter looks on purpose.

The urgency comes from what happens next. A DSPy optimizer does not want your program to be good. It wants your number to go up, and it will find every route to that outcome, including the ones you did not intend to leave open.

A Revolver, Not a Foundation

Mon, 14 Sep 2026 05:00:10 +0100

Part 2 — Get the Model Out of the Chat Box

The model will change; the frame should survive it

The book you could write better next year

Write a book with AI this year and you will be able to write the same book better next year. And better again the year after. The same is true of the code, the research synthesis, the review process, the classifier — anything where a model does part of the work.

The Nearest Neighbor Can Be Wrong

Mon, 07 Sep 2026 11:10:00 +0000

Part III — Retrieval Is an Experiment

The result that is closest and wrong

Illustrative example — not a RELATE measurement.

Query:

Is Dublin the capital of Ireland?

Imagine the passage that ranks first is this one:

Dublin is not, and has never been, the capital of Ireland — that
distinction belongs to the older seat of government at Tara.

Fluent, on-topic, confidently phrased, and among the geometrically closest things in the corpus — and false. A model handed this passage as context may repeat its claim. The retrieval system did its job: it found a near vector. “Near” was not “correct.”

Optimize the Instructions

Fri, 28 Aug 2026 10:55:00 +0100

Chapter 11’s optimizer could only select demonstrations. It never touched an instruction, and it never looked at the development set — which is why upgrading its metric changed nothing.

MIPROv2 does both.

instruction candidates
+
demonstration candidates
+
metric
+
development data
 ↓
candidate program

This is a strictly larger search over a strictly larger space, evaluated against evidence the previous optimizer could not see. If chapter 11’s result was an artifact of a weak mechanism, this is where it gets corrected.

A Resolved Promise Is Not a Correct Answer

Wed, 02 Sep 2026 16:00:00 +0100

The following code has only one definition of success:

try {
 const output = await session.prompt(input);
 show(output);
} catch (error) {
 showFailure(error);
}

If the promise resolves, the interface displays the result.

That is sufficient for a deterministic lookup whose return value is authoritative. It is not sufficient for a probabilistic system that can return fluent, well-formed and incorrect text.

A resolved promise proves that an operation completed according to the runtime contract. It does not prove that the feature did its job.

Evaluate the Feature, Not the Demo

Wed, 02 Sep 2026 16:15:00 +0100

A demo asks whether one carefully chosen example can work.

A feature evaluation asks how often the product contract holds across the inputs, states and failures users will actually encounter.

Both have value. Confusing them is the problem.

1. The unit of evaluation is the feature

“Evaluate Gemma 4” is too broad for our application. “Evaluate this prompt” is too narrow.

The useful unit is something a user attempts:

summarize selected documentation
rewrite permission copy without changing authority
detect language and decide whether to translate
produce structured arguments for a read-only tool

Each feature includes input preparation, capability selection, model output, validation, fallback and UI behavior.

Appendix: The Evidence Ledger

Mon, 31 Aug 2026 09:00:00 +0100

The book argues that a claim should be governed by its provenance, its evidential support, and an explicit statement of what it does not establish. This appendix applies that discipline to the book itself.

Every original empirical number produced for this book is listed here with three things: where it was measured, what its provenance status is, and what it does not establish.

Two facts are visible immediately from the ledger:

Don't Let the Optimizer Cheat

Fri, 28 Aug 2026 11:15:00 +0100

Opening — The Optimizer Doesn’t Have to Cheat Intentionally

In Chapters 15 and 16 we expanded the program’s access. The agent could search repositories, retrieve memories, and adaptively select evidence. Chapter 17 then added search over reasoning states, showing that tree search can allocate inference-time compute toward promising paths.

Now the question becomes urgent: once a program can search, remember, and adaptively select both evidence and reasoning, how do we prove that it did not obtain information it was never supposed to see?

Tool Choice Is a Behavioral Problem

Wed, 02 Sep 2026 17:30:00 +0100

Chapter 18 built a tool that is correct by construction: its contract is tested, its results are bounded, its lifecycle is traced. None of that determines whether an agent uses it well.

Once a model can see several capabilities, the outcome depends on a behavioral pipeline:

discovery → selection → argument construction → admission → execution → result use

A single “task passed” score cannot tell you which stage failed. Chapter 13 made this argument for model output: a resolved promise is not a correct answer. The same layering applies to tool use, with more stages and higher stakes.

Did the Bridge Preserve the Space?

Mon, 07 Sep 2026 14:30:00 +0000

Part VI — Crossing Embedding Spaces · What did each measurement actually certify?

One bridge, eight measurements

Chapter 20 built the machinery that turns preservation evidence into a scoped authorization — a requirement, a measured value, and a decision, all cited together. What it deliberately did not do is ask how much any one of those measured values is actually entitled to say. That is this chapter’s entire job.

A preservation metric is a sensor. Before trusting its reading, ask which failure modes the sensor is even capable of seeing.

Can a Smaller Representation Preserve a Larger One?

Mon, 07 Sep 2026 15:00:00 +0000

Part VII — What Survives Transformation

Two vectors for one document

full_doc → E(full_doc) one vector, dimension d
summary(doc) → E(summary(doc)) one vector, dimension d

Both vectors have exactly the same dimension. What became smaller is the text — the number of words available to encode — not the vector. One embedding was produced from a full document; the other from a much shorter compression of it, passed through the identical embedding pipeline. This is not the dimensionality reduction Chapter 7 covered, and it is not the cross-space translation Part VI built — both the full document and its compression are embedded natively, by the same encoder, into the same space. The question is:

Did the Context Help?

Wed, 23 Sep 2026 05:00:23 +0100

Two prerequisites have been met, on small cases. The compiler builds a legal bundle and can show its working (Chapters 23 and 24). The bundle can be delivered to a running model and independently seen to have arrived (Chapter 25). Neither says anything about the model.

This chapter asks a different question. Given that the intended context was constructed and delivered, did it improve what the model did?

The book now has three kinds of evidence, and this is the third.

How Much of You Can an AI Reproduce?

Fri, 25 Sep 2026 02:45:00 +0100

Part III taught a system to adapt to a person without reproducing them. Now the boundary gets pushed deliberately: given enough evidence — interviews, policies, artifacts, interaction history — how much observable behaviour can an AI reproduce, and how is that claim measured? Prediction, not identity. This chapter measures the first and refuses the second so completely that Chapter 31 inherits a clean, hungry question.

The first discipline is vocabulary. “Reproduce” must not float between style, preferences, values, decisions, and identity. The chapter fixes a reproduction surface of observable targets — language/expression, declared preferences, factual self-knowledge, value judgements, task choices, scenario decisions, multi-step behaviour — governed by one rule:

Advanced Agents From First Principles 23: Your Infrastructure Is Healthy. Why Is the Agent Getting Worse? Detect Behavioral Drift and Roll Back Safely

Sun, 09 Aug 2026 12:00:00 +0100

Your Infrastructure Is Healthy. Why Is the Agent Getting Worse?

Your dashboards are green.

The model endpoint is responding.

The browser workers are alive.

The database is healthy.

The queue is draining.

The verifier service is up.

Latency has not exploded.

There are no obvious exceptions.

And yet the agent is getting worse.

It fixes fewer bugs.

It retrieves weaker evidence.

It escalates to expensive models more often.

It chooses the wrong tools more frequently.

Advanced Agents From First Principles 17: What Is Your Agent Actually Uncertain About?

Sun, 09 Aug 2026 11:02:00 +0100

What Is Your Agent Actually Uncertain About?

An agent reaches a difficult point in a task.

It is not sure what to do next.

A common implementation responds like this:

uncertain
 ↓
call the model again
 ↓
still uncertain
 ↓
call a stronger model
 ↓
still uncertain
 ↓
search more

That is not a reasoning strategy.

It is a spending strategy.

The system is using more computation without identifying what information is actually missing.

Advanced Agents From First Principles 14: Can Your Agent Learn From Its Own Trajectories Without Learning the Wrong Lessons?

Sun, 09 Aug 2026 10:33:00 +0100

An advanced agent now leaves behind something extremely valuable:

evidence.

Not merely chat history.

Not merely model outputs.

Not merely traces.

A sufficiently instrumented system can record:

what state it was in,
what alternatives it considered,
which route it selected,
what branches it pruned,
which model or specialist it escalated to,
which tools it called,
which critic changed the answer,
what verification evidence was produced,
how much compute was spent,
and whether the final result actually passed.

That immediately suggests a tempting idea:

Agents From First Principles 09: AI Agent Says It Worked When It Didn’t? Verify the Result Outside the LLM

Sat, 08 Aug 2026 17:31:00 +0100

An AI agent says:

Done. The task is complete.

That sentence is almost worthless.

The agent may have:

edited the wrong file,
changed the right file incorrectly,
skipped part of the request,
broken another subsystem,
failed to save its work,
misread a tool result,
passed a stale test,
inspected the wrong environment,
or simply decided that its own answer looked convincing.

The central problem is simple:

The system that produced the answer should not be the only system deciding whether the answer is correct.