Agent Evaluation on AIBussin — AI applications, systems and books

Candidate Generation and Selection

Sat, 08 Aug 2026 15:54:00 +0100

The previous chapter built a boundary that stops arbitrary model output from acquiring execution authority without explicit checks. It made one class of failure inspectable and enforceable, and it is silent about another.

Suppose the model is asked to solve a coding problem. One run produces the right patch. The next produces a plausible but incomplete one. A third produces something better again. Nothing is malformed, nothing violates the action schema, and every one of them would pass the boundary we just built. Validity and quality are different properties. A proposal can be completely valid and still be a poor choice, which relocates the uncertainty rather than removing it:

Evidence and Verification

Sat, 08 Aug 2026 17:31:00 +0100

The agent says:

Done.

That is a claim.

It is not evidence.

Every mechanism in this book so far has made the agent better at deciding what to do, and none of them establishes that the user’s goal was achieved. A planner can produce a coherent plan for the wrong problem. A tool can return exit code zero without producing the intended effect. A search can select the highest-scoring branch when every branch is wrong. A memory system can retrieve a perfectly relevant fact that stopped being true in March.

Advanced Agents From First Principles 12: Is Your Advanced Agent Actually Better? Benchmark It Under Equal Budgets

Sun, 09 Aug 2026 10:10:00 +0100

Is Your Advanced Agent Actually Better? Benchmark It Under Equal Budgets

You replace one model call with eight.

Success rises from 62% to 74%.

Great.

Except the new system used:

eight times the inference,
three extra judges,
two rounds of critique,
a larger context,
a stronger verifier,
and several times the latency.

Did the architecture improve?

Or did you just buy more attempts?

This is one of the easiest mistakes to make in advanced agent engineering.

Advanced Agents From First Principles 11: Which Advanced Agent Architecture Should You Use? A Practical Selection Guide

Sun, 09 Aug 2026 09:20:00 +0100

Which Advanced Agent Architecture Should You Use?

You now have too many options.

That is a better problem than having none.

But it is still a problem.

You can add:

self-consistency,
Tree of Thoughts,
beam search,
Monte Carlo Tree Search,
evolutionary search,
specialist routing,
planner/executor/critic separation,
multi-agent debate,
adaptive policies,
learning from previous runs,
or a mixture-of-agents runtime that chooses among several of them.

The temptation is to combine everything.

Agents From First Principles 02: AI Agent Gives Inconsistent Answers? Generate Multiple Candidates and Rank Them

Sat, 08 Aug 2026 15:54:00 +0100

One of the first things you notice when you build anything around a large language model is that the same prompt does not always produce the same quality of answer.

Sometimes the first response is excellent.

Sometimes it is merely acceptable.

Sometimes it misses the point entirely.

That creates a very common agent-engineering question:

If the model is inconsistent, should the agent trust the first answer it gets?

Often, no.