← Pi Agents

Testing Stochastic Software

Unit tests, structural tests and evaluations answer three different questions about code that contains a model; this chapter builds all three against the claim checker and shows why the average success rate hides the number that matters.

Pi Agents Chapter 41 of 46

01 02 03 04 05 06 07 08 09 10 11 12 13 14 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 A B C

Colab

Example

Measure agent reliability with pass@1 and pass^k

Open standalone example →

Three activities all get called “testing”: scripted (a response you chose), structural (assert on events, not wording) and evaluation (how often the whole thing works over repeated runs). This page is the evaluation harness — the measuring parts of eval.ts, excerpted — on Pi 1.0.4. The “model” below is a seeded 85% coin, so this measures the harness’s arithmetic, not any provider; the ledger contains no real-model evidence.

Run it

The chapter’s canonical eval.ts is a self-contained file (types included); these are its load-bearing pieces. evaluate() runs every task k times, counts a thrown run as a failure, and returns the two rates:

export async function evaluate<T>(tasks: Task<T>[], k: number, options: EvalOptions = {}): Promise<EvalResult> {
	if (!Number.isInteger(k) || k < 1) throw new Error(`k must be a positive integer, got ${k}`);
	if (!tasks.length) throw new Error("evaluate needs at least one task; an empty list has no rate to report");
	const guard = options.timeout ?? defaultTimeout;
	const budget = options.timeoutMs;
	// Spread the budget over the runs, with a floor, so one run cannot consume the
	// whole evaluation's allowance.
	const perRun = budget ? Math.max(1, Math.floor(budget / (tasks.length * k))) : undefined;

	let single = 0;
	let allK = 0;
	for (const task of tasks) {
		let wins = 0;
		for (let i = 0; i < k; i++) {
			try {
				const result = perRun ? await guard(() => task.run(), perRun) : await task.run();
				if (task.passes(result as T)) wins++;
			} catch {
				// A step that throws, or times out, is a failed run. Not a crashed
				// evaluation: the point of pass^k is that a run which did not work
				// counts against the task, and a timeout is exactly that.
			}
		}
		single += wins / k;
		if (wins === k) allK++;
	}
	return { tasks: tasks.length, k, passAt1: single / tasks.length, passPowK: allK / tasks.length };
}

export async function evaluate<T>(tasks: Task<T>[], k: number, options: EvalOptions = {}): Promise<EvalResult> {
	if (!Number.isInteger(k) || k < 1) throw new Error(`k must be a positive integer, got ${k}`);
	if (!tasks.length) throw new Error("evaluate needs at least one task; an empty list has no rate to report");
	const guard = options.timeout ?? defaultTimeout;
	const budget = options.timeoutMs;
	// Spread the budget over the runs, with a floor, so one run cannot consume the
	// whole evaluation's allowance.
	const perRun = budget ? Math.max(1, Math.floor(budget / (tasks.length * k))) : undefined;

	let single = 0;
	let allK = 0;
	for (const task of tasks) {
		let wins = 0;
		for (let i = 0; i < k; i++) {
			try {
				const result = perRun ? await guard(() => task.run(), perRun) : await task.run();
				if (task.passes(result as T)) wins++;
			} catch {
				// A step that throws, or times out, is a failed run. Not a crashed
				// evaluation: the point of pass^k is that a run which did not work
				// counts against the task, and a timeout is exactly that.
			}
		}
		single += wins / k;
		if (wins === k) allK++;
	}
	return { tasks: tasks.length, k, passAt1: single / tasks.length, passPowK: allK / tasks.length };
}

seeded() gives a simulated model deterministic pseudo-randomness, so the evaluation is repeatable — a real run would call the provider instead:

// Deterministic pseudo-randomness, so an evaluation of a *simulated* model is repeatable.
export function seeded(seed: number): () => number {
	let a = seed >>> 0;
	return () => {
		a = (a + 0x6d2b79f5) >>> 0;
		let t = a;
		t = Math.imul(t ^ (t >>> 15), t | 1);
		t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
		return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
	};
}

// Deterministic pseudo-randomness, so an evaluation of a *simulated* model is repeatable.
export function seeded(seed: number): () => number {
	let a = seed >>> 0;
	return () => {
		a = (a + 0x6d2b79f5) >>> 0;
		let t = a;
		t = Math.imul(t ^ (t >>> 15), t | 1);
		t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
		return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
	};
}

Expected outcome

testing.test.ts builds 40 tasks from one simulated model that is right 85% of the time, runs each k = 5 times with seed 42, and pins two numbers (observed):

pass@1 = 0.86
pass^5 = 0.45

pass@1 = 0.86
pass^5 = 0.45

pass@1 is the mean single-run rate — what a demo reports. pass^5 is the fraction of tasks that succeeded on all five runs — what a service needs. For independent runs with probability p each, all-k is p^k: 0.85⁵ ≈ 0.44, exactly what the simulation produced. Reliability compounds downward, and the average hides it. A step that throws (or times out) is a failed run, not a crash: the second evaluation test asserts the run count stays at 5 after a throw.

Mechanism and limitations

Documented: the faux provider and its response factories (pi-ai README and declarations, 1.0.4). Observed: the four tests in testing.test.ts — in particular the two pinned figures, which are properties of the simulation, not of any real model. Proposed: the three-level distinction, the rules for a meaningful evaluation (record model and date, use cases you did not tune on, include the failures that matter), and the timeoutMs budget spread across the runs.

Replacing the seeded run with a real provider is where the credential and the bill come in; neither exists here, and nothing estimates a real model’s pass^k.

Full source and test .

The tests in chapters 36 to 40 all passed. They tell you less than you might hope, and it is worth being exact about what.

They show how your code handles the model responses you scripted. They cannot establish how often a real model will produce those responses. A scripted provider can simulate refusals, rambling answers and wrong tool calls, but only when you write those cases into the script. If you stop at green tests, you have not yet measured the stochastic part.

Three different activities get called “testing” here, and they answer different questions. Keeping them apart is the whole chapter.

1. Scripted test      Does my code handle this response correctly?
2. Structural test    Did the right things happen, regardless of the words?
3. Evaluation         How often does the whole thing work, over repeated runs?

The three levels, and what each can reach

    flowchart TD
  subgraph L1["Level 1 — scripted"]
    S1["script a response<br/>assert your handling of it"] --> R1["free, deterministic, in CI"]
  end
  subgraph L2["Level 2 — structural"]
    S2["run the real loop<br/>assert on events"] --> R2["still free under a script<br/>tolerates wording changes"]
  end
  subgraph L3["Level 3 — evaluation"]
    S3["k repeated runs per real task<br/>with independence assessed"] --> R3["costs requests, money, time<br/>rates have uncertainty"]
  end
  L1 -->|"a case you thought of"| L2
  L2 -->|"a rate you need"| L3
  L3 -.->|"never substitute upward or downward"| X1["levels 1 and 2 never become 3"]

The solid arrows are the escalation path and they only go one way. The dotted arrow is the mistake worth naming: a green level 1 suite is routinely reported as “the agent works”, and the two sentences are about different things.

Level 1: scripted

The faux provider from chapter 36 replays responses you wrote. The test controls the model completely, so the result is exact and repeatable:

faux.setResponses([fauxAssistantMessage([fauxToolCall("submit", { severity: "high" })], { stopReason: "toolUse" })]);
assert.deepEqual(await step(models, model, "Triage.", "x", Triage), { severity: "high" });

This level is unit testing for code that happens to contain a model call. It is the right tool for: does assess() reject a bad schema, does step() stop at maxAttempts, does the pipeline refuse an invented citation. These are properties of your code, and a scripted run tests them without a network, a key or a provider bill. It establishes behavior for the supplied fixtures, not every possible response. It belongs in every CI run.

Its limit is built in. The model said exactly what you told it to say. So a passing scripted test is evidence about your handling of that response, and no evidence at all that the model produces it.

A response can also be a function. fauxProvider accepts a factory that receives the transcript the provider was sent, which is how chapter 39 checked that transformContext changed what went out. Use that to assert on the request, not only on the reply.

One trap: do not write assertions that only restate the script. A test that scripts "supported" and asserts "supported" is testing the faux provider. Assert on what your code did with the answer.

Be equally clear about what the faux provider can and cannot manufacture for you. It will happily script a refusal, a rambling answer, a wrong tool call or a malformed submission — those are the cases worth having in a suite, and they are all cases you chose. It cannot produce an autonomous surprise. Everything a scripted test exercises is behaviour you anticipated and wrote down, which makes these tests excellent at finding regressions in your handling and structurally unable to find a failure mode you did not think of. Level 3 exists for that, and levels 1 and 2 never substitute for it.

Level 2: structural

The second level asserts on what happened, not on the exact words produced. Unlike free-form text, event types and fields have defined contracts; the sequence of events can still vary when a real model chooses different tools or numbers of turns. A structural test runs the real loop under a script and checks those events. Under a script this demonstrates that the assertions are about behavior rather than wording; it does not demonstrate that the same event sequence will occur with a real model. The example below drives a small submit agent built inline, not the claim checker. Tests that no tool outside an allowlist ran, or that a blocked call had no effect, are further useful checks; this chapter does not run those checks (chapter 39’s gate test covers the second one):

assert.ok(events.includes("tool:submit:ok"));
assert.equal(faux.state.callCount, 1, "terminate must save the follow-up request");
assert.equal(events.filter((e) => e === "turn_start").length, 1);

Assert that the right tool ran, that the loop took the expected number of turns, that no disallowed tool executed, and that a blocked call caused no effect. These assertions remain useful if you swap the model, but their results may change: a real model can choose another tool or another number of turns. Carry the checks into level 3 and measure how often the required structural invariants hold, rather than assuming the scripted event sequence will recur.

Request counts are a particularly good structural assertion, because they are a cost and a behaviour at once. “The cheap path makes exactly two requests” is a regression test for a pipeline’s economics.

Level 3: evaluation

Now the question changes. You are no longer asking only whether code handles a specified case. You are asking how often a system with variable model behavior produces an acceptable result. One run is a useful observation, but not a reliable estimate of a success rate. Repeated runs across representative tasks are needed.

// Level 3 of chapter 41: evaluation. A scripted provider cannot tell you how
// often a real model gets it right. Repeated runs estimate both per-run success
// and repeatability; neither metric should be interpreted without the other.

export interface Task<T> {
	id: string;
	run: () => Promise<T>;
	passes: (result: T) => boolean;
}

export interface EvalResult {
	tasks: number;
	k: number;
		/** Mean observed success fraction across the k trials for each task. */
	passAt1: number;
	/** Fraction of tasks that passed on every one of their k trials. */
	passPowK: number;
}

export interface EvalOptions {
	/** Divide this target duration across runs as per-run response deadlines.
	 *  This is not a hard evaluation deadline and does not cancel model requests. */
	timeoutMs?: number;
	/** Injected for tests; defaults to the platform timer. */
	timeout?: (fn: () => Promise<unknown>, ms: number) => Promise<unknown>;
}

const defaultTimeout = (fn: () => Promise<unknown>, ms: number): Promise<unknown> =>
	new Promise((resolve, reject) => {
		const timer = setTimeout(() => reject(new Error(`timed out after ${ms}ms`)), ms);
		// Observe both outcomes, including a late rejection after the deadline.
		// A timeout rejects the wait; it does not stop the underlying operation.
		Promise.resolve().then(fn).then(
			(value) => { clearTimeout(timer); resolve(value); },
			(error) => { clearTimeout(timer); reject(error); },
		);
	});

// Reject inputs that would produce meaningless measurements. Without the guards,
// an empty task list produces NaN; k = 0 produces NaN for passAt1 and 1 for
// passPowK (all zero trials pass vacuously). A bad timeout is also an error.
export async function evaluate<T>(tasks: Task<T>[], k: number, options: EvalOptions = {}): Promise<EvalResult> {
		if (!Number.isInteger(k) || k < 1) throw new Error(`k must be a positive integer, got ${k}`);
	if (!tasks.length) throw new Error("evaluate needs at least one task; an empty list has no rate to report");
	if (options.timeoutMs !== undefined && (!Number.isFinite(options.timeoutMs) || options.timeoutMs <= 0)) {
		throw new Error("timeoutMs must be a positive finite number of milliseconds");
	}
	const guard = options.timeout ?? defaultTimeout;
	const budget = options.timeoutMs;
	// This only allocates a per-run response deadline; setup overhead and work
	// continuing after a timeout can exceed the requested total duration.
	const perRun = budget !== undefined ? Math.max(1, Math.floor(budget / (tasks.length * k))) : undefined;

	let single = 0;
	let allK = 0;
	for (const task of tasks) {
		let wins = 0;
		for (let i = 0; i < k; i++) {
			try {
				const result = perRun ? await guard(() => task.run(), perRun) : await task.run();
				if (task.passes(result as T)) wins++;
						} catch {
				// A thrown error or response deadline counts as a failed trial.
				// This basic harness does not retain the reason. Underlying requests
				// can continue after a timeout unless the caller cancels them.
			}
		}
		single += wins / k;
		if (wins === k) allK++;
	}
	return { tasks: tasks.length, k, passAt1: single / tasks.length, passPowK: allK / tasks.length };
}

// Deterministic pseudo-randomness, so an evaluation of a *simulated* model is repeatable.
export function seeded(seed: number): () => number {
	let a = seed >>> 0;
	return () => {
		a = (a + 0x6d2b79f5) >>> 0;
		let t = a;
		t = Math.imul(t ^ (t >>> 15), t | 1);
		t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
		return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
	};
}

Two numbers come out, and the gap between them is the lesson.

passAt1 estimates per-run success by averaging each task’s observed success fraction over its k trials. It answers a question such as: “How often did a trial pass across this task set?”

passPowK, written pass^k, is the fraction of tasks for which all k observed trials passed. It measures repeatability under this evaluation setup, not the probability that an individual production request succeeds. Runs should be repeated under comparable conditions; the harness does not establish statistical independence or protect against shared state between trials.

The test simulates a model that is right 85% of the time on every task, across 40 tasks and 5 runs each, with a seeded generator so the simulation is repeatable:

pass@1 = 0.86
pass^5 = 0.45

The first number looks healthy; the second says that in more than half of the 40 simulated five-run groups, at least one trial failed. They are not in conflict. If each trial succeeds independently with the same probability 0.85, the probability of five successive passes is 0.85⁵, about 0.44. In real datasets tasks have different success probabilities, so pass^k is generally not calculated by raising the overall pass@1 estimate to the power k. Both rates and their uncertainty need to be reported.

Be clear about what this does and does not demonstrate. The “model” is a scripted function drawing from a seeded random generator. The test demonstrates the arithmetic and the harness, not a real model’s reliability. A real evaluation replaces the simulated run with a real provider call and records any model seed, sampling settings and provider version where available. It still cannot assume the trials are independent merely because the calls were repeated.

Running it against a real model

The change is in how run is built, and nothing else in the harness moves:

import { builtinModels } from "@earendil-works/pi-ai/providers/all";

const models = builtinModels();
const model = models.getModel("anthropic", "claude-sonnet-5")!;

const tasks = cases.map((c) => ({
	id: c.id,
		run: () => assess(models, model, c.claim, c.evidence),
	passes: (r) =>
		r.verdict === c.expected &&
		r.citations.every((citation) => c.evidence.includes(citation)) &&
		(r.verdict === "insufficient" || r.citations.length > 0),
}));

const result = await evaluate(tasks, 5);

Documented-API, not run by this book’s suite, since it needs a credential and may incur provider charges. The example checks the expected verdict and basic citation membership; those checks do not prove that the cited evidence supports the conclusion. Six rules make a real evaluation more meaningful:

Record what you measured. The model identifier and version, the date, k, the number of tasks, and the pinned Pi release. A pass^k with no model attached is a number with no meaning.
Use cases you did not tune on. If you edited the prompt until your twenty examples passed, you have measured your prompt’s fit to those twenty.
Include the failures that matter. The invented citation, the contradictory evidence, the claim with no relevant evidence at all. A set made only of easy cases measures nothing you will be hurt by.
Count a thrown step as a failed run. evaluate() does, and a test now exercises that branch: a run that throws is counted as a failure and the evaluation carries on. (The first version’s simulated model never threw, so this branch was untested, which an independent review caught.) StepFailed.kind can distinguish provider failures from noncompliant model output, but this implementation discards that distinction in its catch block. A production evaluation should retain error kinds and timeout counts alongside the aggregate success rates.
Validate the inputs, because the failure is silent. evaluate() rejects an empty task list and a k that is not a positive integer. Without those guards, an empty list produces NaN for both rates; with k = 0, pass@1 becomes NaN while pass^k becomes 1, because every task vacuously passes all zero trials. The revised example also rejects a nonpositive or non-finite timeoutMs. The originally reported tests cover the empty-list and invalid-k guards; the timeout validation should receive its own regression tests.
Bound execution and cost separately. The harness schedules tasks × k trials, not necessarily that many provider requests: a trial can contain several calls. Its timeoutMs option divides a target duration into per-trial response deadlines, counting late responses as failures. It is not an absolute wall-clock budget and does not cancel the provider operation. For unattended evaluations, propagate an abort signal through run, enforce provider/request cancellation and track actual calls, tokens, costs and the overall deadline. Those additional guarantees are not implemented or tested by this example.

The three levels together

	Scripted	Structural	Evaluation
Question	Does my code handle this response?	Did the right things happen?	How often does it work?
Model	faux, fully scripted	faux or real	real
Deterministic	yes	yes under a script	no
Cost	none	none under a script	requests, money, time
Runs	every commit	every commit	on a schedule, and when the model or prompt changes
A failure means	a mismatch with the expected fixture behavior	the observed run violated an invariant	a failed trial under the chosen acceptance criteria
A pass means	your code handles that fixture	the structural assertion held in that run	an estimated rate on this dataset, with uncertainty you must report

The common failure is substitution: treating level 1 as level 3. A suite of scripted tests that all pass says “the plumbing is correct”, and teams hear “the agent works”. The two sentences are about different things, and only the second is about the model.

What the numbers do not say

Forty tasks by five runs means 200 trials; in the simple simulation each trial is one draw, but a real pipeline trial may use several requests. Here the “forty tasks” are one simulated task repeated forty times. The figures (0.86 and 0.45, pinned in the test) therefore describe one 85% coin, not performance on forty distinct problems. A representative task set can have very different, uneven per-task success rates. With only 40 task groups, pass^5 also has substantial sampling uncertainty. Small differences between prompts may be noise; use more representative tasks, uncertainty intervals and preferably paired comparisons before treating an observed difference as an improvement.

A pass means only that the result satisfied the supplied passes predicate. In the real-model snippet that includes the expected verdict and a limited citation membership check; it does not check whether the evidence entails the verdict. Your results are only as good as the task labels, acceptance criteria and review of ambiguous cases. Where several answers are acceptable, passes must encode that.

And a result describes a model at a time. Providers change models behind a stable name. A pass^k from last month is a statement about last month, which is why the record in the first rule above is not bookkeeping.

The external idea

pass^k as a way to measure agent reliability is not this book’s invention. The reference is in Appendix A, which also flags that the figures quoted from that work were not independently verified here. The definition used above stands on its own: it is the code in eval.ts, and the arithmetic is checkable by hand.

What you can and cannot claim

Documented: the faux provider and its response factories (pi-ai README and declarations, 1.0.4).

Observed: the four tests in examples/ch41-testing/. In particular that the simulated 85% model yields pass@1 of 0.86 and pass^5 of 0.45 over 40 tasks, with seed 42, on 1.0.4. That figure is a property of the simulation. It is not a measurement of any real model.

Proposed: the three-level distinction, the rules for a meaningful evaluation, and the schedule in the table. These are practice, not Pi’s contract.

You have a claim checker that is typed, composed and tested under scripted responses, plus an evaluation harness for measuring it against a real provider. The book has not yet measured the claim checker’s real-model reliability. What you have not asked is what the checker is allowed to affect. It reads a corpus. If it could also write one, send a message or run a command, none of the five chapters before this would tell you whether it should.

The last chapter is about the difference between what an agent can do and what it may.