Uncertainty on AIBussin — AI applications, systems and books

When Models Make Things Up

Sat, 29 Aug 2026 22:10:00 +0100

A language model can produce an answer that is fluent, grammatical, detailed, confident, and wrong.

That combination is what makes hallucination interesting.

Ordinary software failures usually leave evidence. A parser throws an exception. A database rejects an invalid constraint. A test fails. A network call times out. The failure changes the shape of the result.

A hallucinating language model can do something more difficult to handle: it can fail without looking broken.

Hallucination Is Not One Thing

Sat, 29 Aug 2026 22:45:00 +0100

Chapter 1 began with a familiar failure: a language model says something fluent that it cannot justify.

That description is useful until we try to build a detector.

Consider six outputs:

A. The model invents a person who does not exist.

B. The model names two real companies but reverses which acquired which.

C. The model cites a real paper that does not support the claim.

D. The model says it inspected a file that it never received.

E. The model answers a question even though the available evidence is insufficient.

F. The model gives the same generic recommendation after the decisive facts
   of the problem have been reversed.

All six can produce plausible text.

Language Detection Is a Decision, Not an Answer

Wed, 02 Sep 2026 14:15:00 +0100

Translation appears to be a two-step browser-AI pipeline:

text
  ↓
detect language
  ↓
translate to target language

The implementation is short enough to fit inside one event handler.

const candidates = await detector.detect(text);
const sourceLanguage = candidates[0].detectedLanguage;
const translator = await Translator.create({
  sourceLanguage,
  targetLanguage: "en"
});
const output = await translator.translate(text);

The dangerous line is not the model call.

It is the array index.

candidates[0] is the highest-ranked language returned by the detector. That does not make it correct, supported for translation, or confident enough to justify an automatic action.

Knowing When Not to Answer

Sun, 30 Aug 2026 11:15:00 +0100

Chapter 10 ended with an ambiguity.

A model gave weak, generic advice.

Perhaps it ignored decisive evidence.

But perhaps the evidence never justified a decisive answer in the first place.

Those are not the same failure.

Consider this question:

What was Company A's Q3 revenue?

and this evidence:

Q1 revenue was $41.2 million.
Q2 revenue was $43.8 million.
The company expects second-half demand to improve.

A language model can easily produce something that sounds reasonable:

Advanced Agents From First Principles 18: What Should Your Agent Observe Next? Use Expected Value of Information

Sun, 09 Aug 2026 11:06:00 +0100

What Should Your Agent Observe Next?

Your agent is uncertain.

That does not tell you what to do.

In the previous post we split uncertainty into operational categories:

interpretation uncertainty,
evidence uncertainty,
route uncertainty,
state uncertainty,
tool uncertainty,
candidate uncertainty,
verification uncertainty.

That is already better than one generic confidence score.

But it still leaves a harder question:

Which piece of information is worth buying next?

Suppose a coding agent is trying to fix a failing test.

Advanced Agents From First Principles 17: What Is Your Agent Actually Uncertain About?

Sun, 09 Aug 2026 11:02:00 +0100

What Is Your Agent Actually Uncertain About?

An agent reaches a difficult point in a task.

It is not sure what to do next.

A common implementation responds like this:

uncertain
   ↓
call the model again
   ↓
still uncertain
   ↓
call a stronger model
   ↓
still uncertain
   ↓
search more

That is not a reasoning strategy.

It is a spending strategy.

The system is using more computation without identifying what information is actually missing.

The Space Between Models Has Holes: Mapping the AI Gap

Wed, 22 Oct 2025 20:30:36 +0100

🌌 Summary

What if the most valuable insights in AI evaluation aren’t in model agreements, but in systematic disagreements?

This post reveals that the “gap” between large and small reasoning models contains structured, measurable intelligence about how different architectures reason. We demonstrate how to transform model disagreements from a problem into a solution, using the space between models to make tiny networks behave more like their heavyweight counterparts.

We start by assembling a high-quality corpus (10k–50k conversation turns), score it with a local LLM to create targets, and train both HRM and Tiny models under identical conditions. Then we run fresh documents through both models, collecting not just final scores but rich auxiliary signals (uncertainty, consistency, OOD detection, etc.) and visualize what these signals reveal.