Calibration on AIBussin — AI applications, systems and books

How to Evaluate a Hallucination Detector

Sat, 29 Aug 2026 23:14:00 +0100

Chapter 5 gave us something concrete enough to evaluate.

Hallucination Energy has:

a target property
an observable proxy
a mathematical definition
an implementation
continuous outputs
natural-task signal
adversarial failures

That sounds like progress.

It is also the point where it becomes dangerously easy to fool ourselves.

Suppose a detector reports:

AUC = 0.75

Is that good?

Perhaps.

But we still do not know:

which class was called positive
how hard the negatives were
whether train and test shared source documents
whether they shared entities or semantic neighborhoods
what prevalence the deployment actually has
whether the labels were trustworthy
how wide the confidence interval was
what happens at 1% false acceptance
how uncertain that 1% estimate is
whether the threshold transfers to another domain
whether the score is a probability at all
whether the verifier itself failed upstream
whether the generator changed
whether decoding parameters changed
whether the model collapses on structural adversaries

A single number cannot answer those questions.

The First Token

Wed, 07 Oct 2026 09:00:00 +0100

Chapter 4 fixed a world with 77 labels and put a number on it. Chapter 5 loosened one joint: the label set could be described at run time instead of trained in. Both chapters compared providers on accuracy. This one asks a question that comes before any provider runs.

Here is the shortest version of it. A decision model that returns a probability over a set of options must, somehow, turn “which of these?” into a number. The most common way is to write the options into a prompt and read the model’s distribution over whatever comes next. If the options are printed as letters, that is a probability over 26 possible answers. If they are printed as words, it is a probability over strings of different lengths. If they are printed as numbers, it is a probability over digits — and digits are not the same size in every tokenizer. The third-party account of how Jev works that this chapter set out to rebuild (research/primary-sources.md, Victor Dibia’s write-up — not vendor documentation; the vendor has published no internals) describes exactly this: a prompt that ends where the answer goes, then each option scored by its log-probability, then a softmax across options. That is the mechanism. It is also, as we are about to see, at least four different mechanisms depending on how the options are printed.

Confidence Is Not Probability

Wed, 07 Oct 2026 19:00:00 +0100

You have a program that can approve a request or send it to a person. The provider returns a valid distribution, the selected option has a large probability, and your next line is if p > 0.9. What does that comparison buy you?

The Decision Contract checks that an answer covers the requested options and that inability is explicit. It cannot check how often a prediction is correct. You need observations for that second question. This chapter measures the available providers on CPU. It fits no calibration method; Calibration owns that work.

Calibration

Wed, 07 Oct 2026 22:00:00 +0100

You have a queue of requests and a program that branches on a provider’s score. Confidence Is Not Probability left you a warning: a valid distribution can be uninformative, badly scaled, or unreliable after the input distribution changes. You can collect labels and fit a correction. How many labels buy useful evidence, and what does your program need to remember?

The specification still defines the decision and its answer set. The provider still supplies scores. This chapter adds a fitted map and an evidence record. The runtime checks that they belong together. Your application chooses an action; neither a transformed score nor a provenance field chooses its consequences.

Calibration

Mon, 07 Sep 2026 12:10:00 +0000

Part IV — Measuring the Representation

What is 0.81?

Illustrative scenario — invented values for intuition, not a measured result.

A pipeline decides two documents are “duplicates” if their cosine exceeds 0.8. Someone picked 0.8 because it looked reasonable. Imagine the two populations that number is actually sitting between:

distribution of cosine for KNOWN duplicate pairs:      mean ~0.79
distribution of cosine for KNOWN non-duplicate pairs:  mean ~0.71

If those two populations overlap the way this sketch implies, the threshold sits inside both: a true duplicate at 0.78 is rejected, an unrelated pair at 0.82 is accepted. This chapter’s actual experiment, later, replaces the sketch with measured positive and negative score samples and derived discrimination/operating statistics. The artifact does not persist the full distributions themselves, and its 85.6% figure is the coverage of a specific quantile-derived band rather than a generic measure of distributional overlap.

Decisions About Decisions

Thu, 08 Oct 2026 09:00:00 +0100

Decisions About Decisions

Design draft: this chapter argues from the literature and from arithmetic. It runs no provider. Every number below is a consequence of a constructed contingency table or an assumed conditional table, and the assumptions are swept and stated next to the results. None of it is evidence about any model. The provider experiment is deferred.

The problem

Chapter 16 kept decide a pure function: it takes a state, asks a provider, returns a typed outcome. It declined to say how one decision feeds another. Chapter 22 showed why that matters: a filter decides which candidates a later decision reads, and its errors become the later decision’s errors.

Beyond Hallucination Energy: A Three-Dimensional Framework for Reliable AI Outputs

Wed, 22 Apr 2026 10:35:46 +0100

🧩 1. TLDR

AI doesn’t just hallucinate. Sometimes it gives answers that are fluent, safe… and completely useless.

Most discussions about AI failure focus on hallucination:

making things up
getting facts wrong
fabricating sources

That’s real. It matters.

But it’s not the most dangerous failure mode in production systems.

There is a quieter one.

A more subtle one.

And in practice a more pervasive one.

AI systems often fail not by being wrong, but by failing to think at all.

Hallucination Energy: A Geometric Foundation for Policy-Bounded AI

Thu, 05 Feb 2026 13:29:55 +0000

🚀 Summary

This post presents the current research draft and implementation of a geometric framework for bounding stochastic language models through deterministic policy enforcement.

The central contribution is a scalar metric termed Hallucination Energy, defined as the projection residual between a claim embedding and the subspace spanned by its supporting evidence embeddings. This metric operationalizes grounding as a measurable geometric quantity.

We proceed in three stages:

Formal Definition a draft manuscript introducing Hallucination Energy, its mathematical formulation, and its role within a policy-controlled architecture.
Empirical Evaluation structured calibration and adversarial stress testing across multiple domains to assess the robustness and limits of projection-based grounding.
Applied Validation large-scale evaluation on 10,000 samples from the HaluEval summarization benchmark, demonstrating that projection-based containment functions as a strong first-order grounding signal in a real generative setting.

This work does not claim to solve hallucination. Rather, it characterizes the boundary of projection-based grounding, establishes its suitability as a deterministic policy scalar, and documents both its strengths and its structural limitations.

The Space Between Models Has Holes: Mapping the AI Gap

Wed, 22 Oct 2025 20:30:36 +0100

🌌 Summary

What if the most valuable insights in AI evaluation aren’t in model agreements, but in systematic disagreements?

This post reveals that the “gap” between large and small reasoning models contains structured, measurable intelligence about how different architectures reason. We demonstrate how to transform model disagreements from a problem into a solution, using the space between models to make tiny networks behave more like their heavyweight counterparts.

We start by assembling a high-quality corpus (10k–50k conversation turns), score it with a local LLM to create targets, and train both HRM and Tiny models under identical conditions. Then we run fresh documents through both models, collecting not just final scores but rich auxiliary signals (uncertainty, consistency, OOD detection, etc.) and visualize what these signals reveal.