The Models Were Different. Their Mistakes Weren't

Part 5 — More Intelligence Is Not Automatically Better

Measure variety by what gets solved

Reach for a second model and the reasoning feels obvious. Different training, different weaknesses, so where one fails another may succeed. Add a third and you have a portfolio. It is an appealing story, and it is a hypothesis rather than a property of model names.

What you actually need is complementary errors: cases where one generator fails and another succeeds. Different names do not guarantee that. Models trained on overlapping data, prompted the same way, on the same task, can be wrong in the same way. When they are, a portfolio costs more and covers nothing extra.

more model names  ≠  more candidates  ≠  more coverage

Settling that takes a measurement with two rules. Compare like with like: a three-model portfolio has to beat three draws from one model, not one draw. And count what got solved, not how many candidates passed, because those are different denominators answering different questions.

By the end of this chapter you will be able to test a portfolio honestly: matched candidate counts, coverage and candidate pass rate kept apart, rescues counted as set differences, and the result read against the headroom the evaluation actually left. That last part matters here, because this chapter’s measurement ran into a ceiling — which is the problem Chapter 25 sets out to fix.

Seven repairs, one unresolved bug

Open on the stubborn task, before any percentage. The problem statement asks for a function that returns the mean of the list it is given, on every call. The starter code caches its first answer and never lets go:

def average(xs):
    if not hasattr(average, '_n'):
        average._n = len(xs)
        average._s = sum(xs)
    return average._s / average._n

The reference repair is one line: return sum(xs) / len(xs). Delete the state.

Seven candidate repairs were generated across three arms, and none deleted the state. Reading them from the frozen rows: ¹

The draws varied, the models varied, and so did the code, but one idea ran through all seven: the state belongs to the function and needs fixing rather than removing. That is the chapter’s question in concrete form. Did changing model identity expose a solution that repeated baseline sampling did not? On this corpus, no.

Did changing models add verified coverage beyond repeated sampling?

More names, more candidates, more coverage

Count solved tasks, not models. Two denominators must never be substituted for one another:

In P1 the C1 and H1 arms happen to score 0.9167 on both — 33 of 36 candidates, 11 of 12 tasks — and that coincidence is itself a trap. The fractions share a value and answer different questions.

A third boundary rides alongside: the oracle is not a deployable selector. When an arm generates three candidates and one passes, oracle@3 counts the task as covered because the experiment’s checker identifies the passing candidate after the fact. That does not show that a production system could pick it without the checker. Chapter 25 needs this boundary badly; here it stays attached to every coverage sentence. ²

A fragment executed as arithmetic, not as evidence, showing why the denominators diverge in general even when they coincide in P1: ³

candidates = {
    "task-a": [True, True, False],
    "task-b": [False, False, True],
    "task-c": [False, False, False],
}
candidate_passes = sum(p for draws in candidates.values() for p in draws)   # 3
task_coverage = sum(any(draws) for draws in candidates.values())            # 2
# 3/9 candidates pass, but 2/3 tasks are covered.

The experiment design

The comparison is a matched triple. C0 is one draw from the baseline model. C1 is three draws from the same baseline model. H1 is one draw each from three models, the baseline plus two others. The comparison that matters is H1 against C1, because candidate count is matched: C1 asks what three same-model draws buy, and H1 asks what model variety buys on top of that. ⁴

Interpretation was preregistered before any model call. Rule 2 covers what happened: if the portfolio roughly ties the homogeneous arm, keep repeated sampling as the simpler default and look for task-specific unique rescues. Rule 5 itself is broader: if both C1 and H1 remain poor, investigate decomposition, context, task framing, or stronger generators before adding collaboration machinery. The frozen P1 report applied that instinct to the single task every arm failed; that task-level application is an interpretation of the rule, not the preregistered condition itself. ^{5 4}

Two parts of the preregistered design did not survive contact with the environment, and both change what P1 can say. The preregistration named H1 as the baseline model plus Claude and GPT. Cloud keys were unavailable, so the run used two local models instead: mistral:7b-instruct and llama3.1:8b. P1 therefore tested variety among three small local models, not the cross-vendor variety the design anticipated. And the companion money-matched framing was never realizable, because local models have no measured monetary cost; token and latency budgets governed instead. ^{5 4}

How the comparison was run

The corpus is seeded-code-v1: twelve seeded repair tasks, each with starter code, a reference solution, and hidden tests the model never sees. Arms run through sealed fan-out, each candidate is materialized in a workspace, and each hidden-verifier outcome is recorded per candidate. In this chapter, a task is counted as solved when at least one candidate receives the corpus verifier’s PASS. That is the experiment’s operational endpoint, not ground truth beyond the properties those hidden tests exercise. ^{6 1}

The metric definitions live in one place, and the recomputation reuses them. Oracle coverage is the share of attempted tasks with at least one passing candidate. Candidate rate is passes over generated candidates. A rescue is a set difference: tasks solved by one arm and by no other. Conditional failure is P(B fails | A fails) over jointly attempted tasks. The heterogeneity premium is H1’s coverage minus C1’s.

The analysis emits its frozen tiny-n warning at fewer than 30 tasks: do not infer causal superiority. That warning does not turn n=30 into a universal statistical threshold; it is the experiment’s own interpretation rule. Oracle@k is best-of-k with an oracle selector, not deployable performance. ⁷

No runtime change accompanies this chapter; the recomputation equals the historical report exactly.

Two outside results explain the design without contributing evidence to it. Sampling one model repeatedly is the baseline any heterogeneous portfolio has to beat; self-consistency shows how much repeated sampling of one model can buy on its own (Wang et al., 2023). And classifier-ensemble research has long found that diversity among members does not straightforwardly produce ensemble accuracy — useful differences have to be measured, not inferred from distinct labels (Kuncheva, 2003). That is why P1 matched draws first and counted rescue sets rather than model names.

The frozen run, reconstructed

Eighty-four calls over twelve tasks — 12 baseline draws, 36 homogeneous redraws, 36 portfolio draws — produced 84 candidates and 83 checks: mistral’s syntax error never reached the hidden tests. The exports are byte-pinned with verified hashes, the analysis recomputes arm tables from candidate and call rows, and an independent verifier asserts the figures and catches a seeded corruption. ^{2 1}

The three arms solved the identical eleven tasks. The failed-task set is the same single task everywhere, which is why every pairwise conditional failure reads 1.0: whenever one arm failed a task, the other failed it too — because there was only one task to fail. That number describes twelve tasks. It is not evidence that model errors are generally correlated, and the chapter’s title must not do that work.

The candidate rows make the ceiling even flatter than the table suggests. On the eleven easy tasks, all 77 candidates passed — every draw, from every model, in every arm. On the twelfth, all seven failed. There is no per-draw variation anywhere in this run for a portfolio to exploit, in either direction. ¹

Unique rescues in either direction: none. Premium: 0.0. ²

Resources differed while outcomes did not. Tokens ran 1,827 for C0, 5,452 for C1, and 6,549 for H1 — the portfolio at 1.20× the matched homogeneous arm. Reported median call latency was 2.9 s, 2.8 s, and 6.8 s. Tokens, latency, and money stay separate columns: money was never measured, and a median per-call latency is not portfolio wall-clock time. The supported cost sentence is only that H1 spent more tokens for the same verified coverage. ²

The run in one compact table, read from the frozen report:

Heterogeneity premium (H1 − C1): 0.0. Coverage is identical down the column while tokens rise across it — the table is the result. ²

The ceiling

C0 solved eleven of twelve tasks with a single draw. That fact dominates the run. With exactly one task of headroom, the portfolio gained none:

baseline coverage     = 11/12
maximum possible gain = 1 task
observed gain         = 0

P1 still stands as a measurement. With C0 already covering 11 of 12 tasks and no per-draw variation on those eleven, this run left only one task on which the arms could differ. The observed tie therefore has little discriminating force; it does not establish that the interventions are equivalent. The appropriate next experiment is a harder corpus with more room for paired differences, which is why Chapter 25 exists.

In the checker the same ceiling appears. The hidden test for the stubborn task begins with assert average([1, 2, 3]) == 2 or True, an assertion that cannot fail; it exists only to seed stale state. The two assertions after a module reload do the discriminating. They are sufficient to reject the stale-state repairs observed in this fixture, which is exactly the property this result needs. They do not establish that the hidden test is a complete specification of every correct average implementation — Chapter 21’s adequacy boundary still applies. ⁸

An earlier plan for this book imagined a “Council” — reviewer, editor, brand guardian — improving a tutorial through complementary perspectives. That was a belief about variety, not a measured improvement, and it described prompted roles rather than independent sources. P1 replaces the assumption with a checkable question and returns a bounded answer: on this corpus, variety of labels bought nothing. ⁴

What this is not

• Not proof that heterogeneity never helps. Twelve tasks, three local models, one task of headroom.
• Not the experiment that was preregistered. Claude and GPT were replaced by local models; cross-vendor variety was not tested.
• Not equivalence of models. Same coverage establishes the same verified set, not equal ability.
• Not a collaboration finding. No debate or synthesis was tested; Rule 6 forbids conclusions about it.
• Not an economic comparison. No money was measured.

Where it is still weak

1. Ceiling-bound. One task of possible gain, and no per-draw variation on the other eleven. ¹
2. Tiny-n. The analysis marks n=12 below its frozen 30-task threshold and forbids causal-superiority inference from P1; no broader significance claim is established here. ⁹
3. Substituted portfolio. Local mistral and llama stood in for the preregistered cloud models. ⁴
4. Single seeded corpus. Synthetic repair tasks need not resemble any real workload. ¹
5. Unexamined causes. Overlap in training data or prompting behavior was never measured.
6. Oracle, not selector. Coverage counts a task when any candidate passes; nothing chooses among candidates.

Do this now

Thirty minutes. Count solved tasks, not model names.

1. Take any multi-candidate result — three drafts, three reviews, three runs. Mark each candidate pass or fail with a check you trust, and compute both denominators.
2. Build the task-by-arm table. Tasks where every candidate passes are your ceiling; they teach nothing about which arm is better.
3. For the tasks every arm failed, read the failed candidates side by side, as the opening table does. Name the shared idea they all got wrong.
4. Compare the portfolio you ran with the one you planned. If anything was substituted, write it into the result.
5. Write the decision in one sentence with the corpus attached: “keep the simpler default on this workload until this harder set shows otherwise.”

If you are building with an assistant:

Measure variety by solved tasks, never by model names. Match candidate
counts before comparing portfolios: one draw, N same-model draws, N mixed
draws. Keep candidate pass rate and task coverage separate, report the
failed-task intersection and read the failed candidates side by side.
Preregister the interpretation, including what a tie means, and record any
substitution of the planned models. Treat the oracle as a retrospective
finder, not a selector. Keep tokens, latency, and money in separate
columns. When the baseline covers nearly everything, name the ceiling and
build a harder corpus.

Failure modes

• Counting names instead of tasks. Three models sounds like more than one until the solved sets come back identical.
• Comparing H1 to C0. The portfolio must beat matched same-model sampling, not a single draw.
• Quoting 33/36 as coverage. Candidate passes repeat on solved tasks.
• Reading 1.0 overlap as law. A complete intersection on one failed task describes the corpus.
• Reporting the planned portfolio. If the models changed, so did the question.
• Arguing with the ceiling. A null under 11/12 baseline coverage indicts the corpus’s power, not the experiment’s honesty.

What this chapter established

• Different model names do not establish complementary errors. Variety is a hypothesis to test, not a property of labels. P1 observed the same single failed task in every arm; with only one task of headroom, that is a corpus-scoped overlap, not a general claim about model-error correlation.
• Match the comparison. A portfolio has to beat the same number of draws from one model. Comparing it with a single draw measures candidate count, not variety.
• Count solved tasks, and keep the denominators apart. Candidate pass rate and task coverage answer different questions, and an oracle that identifies the passing candidate afterwards is not a selector that could find it in production.
• Read a null against the headroom. When the baseline already covers nearly everything, a tie may have little power to distinguish the methods. Name the ceiling and build a harder test rather than reading the null as evidence of equivalence.

What CodeAI measured. H1 matched C1 exactly: 33/36 candidates, the same 11/12 tasks, zero rescues, premium 0.0, at 1.20× the tokens. ² All 77 candidates on the eleven easy tasks passed and all 7 on the stale-state task failed, each by repairing the state instead of removing it, so the run held no variation for a portfolio to use. ¹ The portfolio tested was three local models, substituted for the preregistered cross-vendor set. ⁴ The preregistered Rule 2 keeps repeated same-model sampling as the simpler default until a harder corpus shows otherwise. ⁵

Evidence notes

Independent verification. The analysis verifier asserts the recomputed coverage, candidate, and token figures per arm, requires zero premium with zero rescues for P1, and must reject a mutated copy. The export verifier checks all four frozen bundles’ hashes. The candidate table in the opening and the 77-of-77 ceiling count were read from the frozen rows for this chapter; they are outside both verifiers. ^{2 1}

The limits travel with the numbers. Coverage here means passing the corpus’s hidden verifier, not deployable selection and not correctness beyond what those tests exercise. The analysis emits its own n=12 tiny-n warning against causal-superiority inference. Three local models on one seeded corpus say nothing about other families, sizes, or vendors — and these three may share training data and failure modes that P1 never measured, so the chapter explains nothing about why the overlap is complete. ^{2 4}

Next

The baseline keeps its place on easy ground, and the measurement says exactly that. The response to a ceiling is a harder test: problems difficult enough that methods have room to differ, where a portfolio might cover more while producing worse candidates.

Continue with When Your Benchmark Is Too Easy.

References

• L. I. Kuncheva. That Elusive Diversity in Classifier Ensembles. Proc. IbPRIA 2003, LNCS 2652, pp. 1126–1138. Publication record.
• Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR, 2023. arXiv:2203.11171.

Implementation sources: P1 ran on the P-series CodeAI lineage (seeded-code-v1 corpus; src/codeai/corpus.py: CORPUS_VERSION, seeded_corpus; src/codeai/experiments.py: ArmDef, run_arm; src/codeai/analysis.py: arm_metrics, conditional_failure, unique_rescues, build_report). No CodeAI change accompanies this chapter. The candidate table and ceiling count were read from frozen rows during editing; the starter, reference, and hidden test come from the corpus source. Evidence: experiments/applied-ai/evidence/p-series/ (frozen exports, hashes verified), experiments/applied-ai/evidence/p-series-analysis/ (analysis.json, TABLES.md, analyze_pseries.py, verify_analysis.py), experiments/P1-results.md, and experiments/P1-interpretation.md. Nothing was rerun and nothing frozen was modified.