False Acceptance Rate on AIBussin — AI applications, systems and books

How to Evaluate a Hallucination Detector

Sat, 29 Aug 2026 23:14:00 +0100

Chapter 5 gave us something concrete enough to evaluate.

Hallucination Energy has:

a target property
an observable proxy
a mathematical definition
an implementation
continuous outputs
natural-task signal
adversarial failures

That sounds like progress.

It is also the point where it becomes dangerously easy to fool ourselves.

Suppose a detector reports:

AUC = 0.75

Is that good?

Perhaps.

But we still do not know:

which class was called positive
how hard the negatives were
whether train and test shared source documents
whether they shared entities or semantic neighborhoods
what prevalence the deployment actually has
whether the labels were trustworthy
how wide the confidence interval was
what happens at 1% false acceptance
how uncertain that 1% estimate is
whether the threshold transfers to another domain
whether the score is a probability at all
whether the verifier itself failed upstream
whether the generator changed
whether decoding parameters changed
whether the model collapses on structural adversaries

A single number cannot answer those questions.

Calibration

Mon, 07 Sep 2026 12:10:00 +0000

Part IV — Measuring the Representation

What is 0.81?

Illustrative scenario — invented values for intuition, not a measured result.

A pipeline decides two documents are “duplicates” if their cosine exceeds 0.8. Someone picked 0.8 because it looked reasonable. Imagine the two populations that number is actually sitting between:

distribution of cosine for KNOWN duplicate pairs:      mean ~0.79
distribution of cosine for KNOWN non-duplicate pairs:  mean ~0.71

If those two populations overlap the way this sketch implies, the threshold sits inside both: a true duplicate at 0.78 is rejected, an unrelated pair at 0.82 is accepted. This chapter’s actual experiment, later, replaces the sketch with measured positive and negative score samples and derived discrimination/operating statistics. The artifact does not persist the full distributions themselves, and its 85.6% figure is the coverage of a specific quantile-derived band rather than a generic measure of distributional overlap.

Beyond Hallucination Energy: A Three-Dimensional Framework for Reliable AI Outputs

Wed, 22 Apr 2026 10:35:46 +0100

🧩 1. TLDR

AI doesn’t just hallucinate. Sometimes it gives answers that are fluent, safe… and completely useless.

Most discussions about AI failure focus on hallucination:

making things up
getting facts wrong
fabricating sources

That’s real. It matters.

But it’s not the most dangerous failure mode in production systems.

There is a quieter one.

A more subtle one.

And in practice a more pervasive one.

AI systems often fail not by being wrong, but by failing to think at all.

Hallucination Energy: A Geometric Foundation for Policy-Bounded AI

Thu, 05 Feb 2026 13:29:55 +0000

🚀 Summary

This post presents the current research draft and implementation of a geometric framework for bounding stochastic language models through deterministic policy enforcement.

The central contribution is a scalar metric termed Hallucination Energy, defined as the projection residual between a claim embedding and the subspace spanned by its supporting evidence embeddings. This metric operationalizes grounding as a measurable geometric quantity.

We proceed in three stages:

Formal Definition a draft manuscript introducing Hallucination Energy, its mathematical formulation, and its role within a policy-controlled architecture.
Empirical Evaluation structured calibration and adversarial stress testing across multiple domains to assess the robustness and limits of projection-based grounding.
Applied Validation large-scale evaluation on 10,000 samples from the HaluEval summarization benchmark, demonstrating that projection-based containment functions as a strong first-order grounding signal in a real generative setting.

This work does not claim to solve hallucination. Rather, it characterizes the boundary of projection-based grounding, establishes its suitability as a deterministic policy scalar, and documents both its strengths and its structural limitations.