Testing on AIBussin — AI applications, systems and books

Evidence and Verification

Sat, 08 Aug 2026 17:31:00 +0100

The agent says:

Done.

That is a claim.

It is not evidence.

Every mechanism in this book so far has made the agent better at deciding what to do, and none of them establishes that the user’s goal was achieved. A planner can produce a coherent plan for the wrong problem. A tool can return exit code zero without producing the intended effect. A search can select the highest-scoring branch when every branch is wrong. A memory system can retrieve a perfectly relevant fact that stopped being true in March.

Machine Coding: Building Software with AI

Fri, 28 Aug 2026 10:00:00 +0100

“First, solve the problem. Then, write the code.”

Summary

Machine coding is no longer just asking an AI to write files for you.

That was the early version. You described an app, the model produced a pile of code, you copied it into a folder, ran it, pasted the errors back, and hoped the loop converged.

That workflow taught us something important: conversation can turn into software. But it also hid the real lesson.

Advanced Agents From First Principles 24: How Do You Release Agent Behavior Safely? Add Behavioral Contracts, Compatibility Checks and Promotion Gates

Sun, 09 Aug 2026 12:09:00 +0100

A model change can pass health checks and still break your agent.

A prompt change can preserve output schema and still change which tools the agent chooses.

A router change can improve average cost and silently starve a specialist that is essential for one high-risk workload.

A memory migration can keep every row intact and still change which memories are retrieved.

A verifier upgrade can make every dashboard greener while making the system less trustworthy.

Agents From First Principles 09: AI Agent Says It Worked When It Didn’t? Verify the Result Outside the LLM

Sat, 08 Aug 2026 17:31:00 +0100

An AI agent says:

Done. The task is complete.

That sentence is almost worthless.

The agent may have:

edited the wrong file,
changed the right file incorrectly,
skipped part of the request,
broken another subsystem,
failed to save its work,
misread a tool result,
passed a stale test,
inspected the wrong environment,
or simply decided that its own answer looked convincing.

The central problem is simple:

The system that produced the answer should not be the only system deciding whether the answer is correct.