Verification on AIBussin — AI applications, systems and books

If There's Any Doubt, It's Deterministic

Mon, 14 Sep 2026 05:00:03 +0100

Part 1 — Where You Stand

Seven operations, one of them intelligent

Take the paragraph review from Chapter 1 and write down everything the process actually has to do. Not the prompt — the process.

Select the paragraph to review.
Decide whether it has changed since the last review.
Decide whether there is budget left for a call.
Identify claims in the paragraph that need supporting evidence.
Extract the flagged sentence and its position.
Check that a cited source resolves and that its year matches the bibliography.
Decide whether to write the change to the file.

Six of those have exactly one correct answer, computable without a model. Selection is an index lookup and change detection is a hash comparison. Budget is arithmetic; extraction is parsing. Source checking resolves the request and compares it against the bibliography. The write decision evaluates policy.

Scrum Built the Training Set

Mon, 14 Sep 2026 05:00:04 +0100

Part 1 — Where You Stand

Why checkable work became unusually favorable terrain

An uncomfortable piece of bookkeeping

Software engineering became an unusually fertile early target for language-model automation. The usual explanation is that code is logical and therefore tractable for a machine.

That explanation points at only part of what made software favorable. The account this chapter defends is about the surrounding work records and checks.

Over decades, software teams increasingly worked through issue trackers, version control, code review, automated tests, and iterative planning methods. Not every team used Scrum, and none of these practices guarantees the others, but together they often left behind unusually structured records:

Evidence and Verification

Sat, 08 Aug 2026 17:31:00 +0100

The agent says:

Done.

That is a claim.

It is not evidence.

Every mechanism in this book so far has made the agent better at deciding what to do, and none of them establishes that the user’s goal was achieved. A planner can produce a coherent plan for the wrong problem. A tool can return exit code zero without producing the intended effect. A search can select the highest-scoring branch when every branch is wrong. A memory system can retrieve a perfectly relevant fact that stopped being true in March.

Building the Complete Agent

Wed, 26 Aug 2026 10:00:00 +0100

Every mechanism in this book was argued against a problem chosen to isolate it.

That isolation was deliberate, and it was also a form of protection. The memory chapter picked a task where recall was the bottleneck, held everything else still, and measured the one thing it came to measure. The result is a clean explanation and a weak claim. Nothing in it establishes that the same retrieval policy behaves when a search controller is expanding forty nodes, or when a verifier insists that every piece of evidence carry a state identity the search controller has never heard of.

From Measurements to Policy

Sun, 30 Aug 2026 18:37:00 +0100

Chapter 11 ended with a typed reliability record.

It might say:

containment = PASS
relation_fidelity = PASS
sensitivity = PASS
epistemic_adequacy = ANSWERABLE
provenance = UNVERIFIED

That record describes the candidate.

It still does not authorize the candidate.

To make the distinction concrete, we built a deliberately flawed reference policy that checked containment, structural fidelity, and answerability but forgot to require verified provenance for a high-risk action.

Running the same immutable record through two policy versions produced:

Research: From Paper to Pipeline with AI

Fri, 28 Aug 2026 10:00:00 +0100

“Research is formalized curiosity. It is poking and prying with a purpose.”

Summary

There is still something powerful about picking up a research paper and asking:

What can I build from this?

That question turns reading into motion. A paper stops being a sealed object. It becomes a source of ideas, mechanisms, tests, questions, and possible systems.

But the modern version of this workflow is more disciplined than the old one.

Verification, Repair, and Rejection

Sun, 30 Aug 2026 18:38:00 +0100

Chapter 12 gave the system a control plane.

It can now say:

commitment = HOLD
next_action = VERIFY

or:

commitment = HOLD
next_action = RETRIEVE

or:

commitment = HOLD
next_action = REFINE

or:

commitment = DENY
response_mode = REJECTION

Those are policy decisions.

They are not yet recovery implementations.

The obvious implementation is dangerously tempting:

The answer failed a check.
Ask the model to fix it.

But the model that produced the first unsupported statement can produce a second unsupported statement while “correcting” it.

Machine Coding: Building Software with AI

Fri, 28 Aug 2026 10:00:00 +0100

“First, solve the problem. Then, write the code.”

Summary

Machine coding is no longer just asking an AI to write files for you.

That was the early version. You described an app, the model produced a pile of code, you copied it into a folder, ran it, pasted the errors back, and hoped the loop converged.

That workflow taught us something important: conversation can turn into software. But it also hid the real lesson.

Building Systems That Distrust Their Models

Sun, 30 Aug 2026 21:01:00 +0100

The first chapter began with a simple observation:

A language model can produce a fluent answer
without possessing a mechanism that proves the answer is true.

Fourteen chapters later, that fact has not changed.

The model can still:

invent
misbind
misattribute
ignore decisive context
answer without enough evidence
accept bad retrieval
repair one error by creating another
repeat its own stored mistake

The final architecture does not make those possibilities disappear.

The Agent Cannot Grade Its Own Homework

Mon, 14 Sep 2026 05:00:21 +0100

Part 4 — Make It Safe and Verifiable

Independence, adequacy, and binding

A model saying its work is correct is not verification. Neither, on its own, is running a separate test. A test can run separately and check the wrong thing. It can check the right thing against a version of the artifact that has since changed. And a PASS that nobody can tie to the exact state being accepted is only another claim.

Build a Production AI Agent From First Principles: The Complete Reference Architecture

Sun, 09 Aug 2026 16:09:00 +0100

Build a Production AI Agent From First Principles: The Complete Reference Architecture

We have spent this series adding mechanisms only when a specific failure demanded them.

We started with a model call.

Then we added candidate generation, critique, planning, tool use, memory, search and verification.

Then the system stopped looking like a clever prompt.

It started looking like software.

Then distributed systems problems arrived:

duplicate work,
retries,
leases,
fencing,
backpressure,
dependency failure,
behavioral drift,
release compatibility,
replay,
incident forensics,
SLOs,
authority,
competence,
capability acquisition,
placement,
handoff,
stale state,
stale intent,
commitments,
durable workflows,
transaction recovery,
trust boundaries,
multi-agent coordination,
and finally an explicit control plane.

At this point the architecture is complete enough that adding another isolated mechanism would make the series worse rather than better.

Advanced Agents From First Principles 19: Can Your Agent Explore in Parallel Without Creating Chaos? Use Speculative Execution and Early Cancellation

Sun, 09 Aug 2026 11:10:00 +0100

Advanced Agents From First Principles 19: Can Your Agent Explore in Parallel Without Creating Chaos? Use Speculative Execution and Early Cancellation

A production agent often has more than one useful thing it could do next.

It could:

inspect repository state,
run a targeted test,
retrieve documentation,
ask a second model to critique a candidate,
generate an alternative implementation,
probe an API,
inspect a deployment,
or verify an invariant.

If those actions are independent, executing them one by one can be needlessly slow.

Advanced Agents From First Principles 18: What Should Your Agent Observe Next? Use Expected Value of Information

Sun, 09 Aug 2026 11:06:00 +0100

What Should Your Agent Observe Next?

Your agent is uncertain.

That does not tell you what to do.

In the previous post we split uncertainty into operational categories:

interpretation uncertainty,
evidence uncertainty,
route uncertainty,
state uncertainty,
tool uncertainty,
candidate uncertainty,
verification uncertainty.

That is already better than one generic confidence score.

But it still leaves a harder question:

Which piece of information is worth buying next?

Suppose a coding agent is trying to fix a failing test.

Advanced Agents From First Principles 17: What Is Your Agent Actually Uncertain About?

Sun, 09 Aug 2026 11:02:00 +0100

What Is Your Agent Actually Uncertain About?

An agent reaches a difficult point in a task.

It is not sure what to do next.

A common implementation responds like this:

uncertain
 ↓
call the model again
 ↓
still uncertain
 ↓
call a stronger model
 ↓
still uncertain
 ↓
search more

That is not a reasoning strategy.

It is a spending strategy.

The system is using more computation without identifying what information is actually missing.

Advanced Agents From First Principles 16: Where Should an Agent Spend Its Compute? Build a Dynamic Budget Scheduler

Sun, 09 Aug 2026 10:53:00 +0100

Where Should an Agent Spend Its Compute?

A production agent has a budget whether you designed one or not.

Every model call costs something.

Every search node costs something.

Every tool invocation costs something.

Every verifier costs something.

Every retry adds latency.

Every escalation to a stronger model spends money and time that could have been used somewhere else.

The naive architecture gives every subsystem its own fixed limit:

MAX_STEPS = 20
MAX_SEARCH_NODES = 32
MAX_CRITIC_CALLS = 3
MAX_RETRIES = 4
MAX_VERIFIER_CALLS = 2

That looks safe.

Advanced Agents From First Principles 15: How Do You Optimize an Agent Policy Without Turning It Into Another Black Box?

Sun, 09 Aug 2026 10:49:00 +0100

How Do You Optimize an Agent Policy Without Turning It Into Another Black Box?

By now our advanced agent can do a lot.

It can:

route tasks to different models or specialists,
decide whether to search,
choose a search budget,
decide when to escalate,
invoke critics,
retry or recover,
stop when evidence is strong enough,
learn from verified production trajectories,
and trace the decisions that produced each outcome.

That creates a new problem.

Advanced Agents From First Principles 14: Can Your Agent Learn From Its Own Trajectories Without Learning the Wrong Lessons?

Sun, 09 Aug 2026 10:33:00 +0100

An advanced agent now leaves behind something extremely valuable:

evidence.

Not merely chat history.

Not merely model outputs.

Not merely traces.

A sufficiently instrumented system can record:

what state it was in,
what alternatives it considered,
which route it selected,
what branches it pruned,
which model or specialist it escalated to,
which tools it called,
which critic changed the answer,
what verification evidence was produced,
how much compute was spent,
and whether the final result actually passed.

That immediately suggests a tempting idea:

Advanced Agents From First Principles 12: Is Your Advanced Agent Actually Better? Benchmark It Under Equal Budgets

Sun, 09 Aug 2026 10:10:00 +0100

Is Your Advanced Agent Actually Better? Benchmark It Under Equal Budgets

You replace one model call with eight.

Success rises from 62% to 74%.

Great.

Except the new system used:

eight times the inference,
three extra judges,
two rounds of critique,
a larger context,
a stronger verifier,
and several times the latency.

Did the architecture improve?

Or did you just buy more attempts?

This is one of the easiest mistakes to make in advanced agent engineering.

Advanced Agents From First Principles 11: Which Advanced Agent Architecture Should You Use? A Practical Selection Guide

Sun, 09 Aug 2026 09:20:00 +0100

Which Advanced Agent Architecture Should You Use?

You now have too many options.

That is a better problem than having none.

But it is still a problem.

You can add:

self-consistency,
Tree of Thoughts,
beam search,
Monte Carlo Tree Search,
evolutionary search,
specialist routing,
planner/executor/critic separation,
multi-agent debate,
adaptive policies,
learning from previous runs,
or a mixture-of-agents runtime that chooses among several of them.

The temptation is to combine everything.

Advanced Agents From First Principles 08: Is Your Agent Spending the Same Compute on Every Task? Build Adaptive Agents That Escalate Only When Needed