Measurement on AIBussin — AI applications, systems and books

The Measurement Instrument

Wed, 23 Sep 2026 05:00:02 +0100

Imagine a developer debugging a slow, expensive coding agent. The visible transcript shows eight user messages and eight assistant replies. It looks modest. Then they enable request logging at the harness layer and discover that the eighth model invocation carried tens of thousands of tokens: the project instructions re-sent for the eighth time, tool definitions for tools never called in this session, two file reads that overlap almost entirely, and a history that includes four failed tool calls and their stack traces in full. The user typed a few hundred words in the whole session. The model received a novella.

The Measurement Instrument

Mon, 14 Sep 2026 05:00:02 +0100

Chapter 1 ended with a test. Does the past change what the system does now, and does it change it for the better?

That is easy to state and hard to run. This chapter builds the thing that runs it.

Two words will recur for the rest of the book, and they must not blur together. The memory system is the thing that remembers. The instrument is the apparatus that decides whether it does. They are separate products, and this book builds both.

Intelligence in the Wrong Direction

Mon, 14 Sep 2026 05:00:07 +0100

Part 1 — Where You Stand

Capability without an objective is magnitude without direction

The thing the marketing does not cover

Every vendor will sell you more capability. None of them can tell you whether that capability moves your process toward its objective without a measurement you supply.

A stronger model can execute the wrong objective more effectively just as it can execute the right one more effectively. Capability alone does not tell you whether the gap between what you wanted and what you got narrowed or widened. That is an empirical question.

Token Counts Don't Add Up

Mon, 14 Sep 2026 05:00:13 +0100

Part 2 — Get the Model Out of the Chat Box

A field name is not a unit: normalize at the boundary

Every AI application ends up doing arithmetic on numbers a provider reported. It adds tokens into a budget, divides an invoice by them, compares two models on them, and routes work to whichever looks cheaper per token.

Those numbers arrive in fields with shared names: input_tokens, cached_tokens, reasoning_tokens. The shared names hide different rules. On one major API, cached input is counted inside input_tokens. On another, it is reported beside it. Add the two and the result is wrong even though every number in it is correct. Nothing crashes. The budget, the cost comparison and the routing decision are simply built on a sum that means nothing.

The Models Were Different. Their Mistakes Weren't

Mon, 14 Sep 2026 05:00:24 +0100

Part 5 — More Intelligence Is Not Automatically Better

Measure variety by what gets solved

Reach for a second model and the reasoning feels obvious. Different training, different weaknesses, so where one fails another may succeed. Add a third and you have a portfolio. It is an appealing story, and it is a hypothesis rather than a property of model names.

What you actually need is complementary errors: cases where one generator fails and another succeeds. Different names do not guarantee that. Models trained on overlapping data, prompted the same way, on the same task, can be wrong in the same way. When they are, a portfolio costs more and covers nothing extra.

Did the Context Help?

Wed, 23 Sep 2026 05:00:23 +0100

Two prerequisites have been met, on small cases. The compiler builds a legal bundle and can show its working (Chapters 23 and 24). The bundle can be delivered to a running model and independently seen to have arrived (Chapter 25). Neither says anything about the model.

This chapter asks a different question. Given that the intended context was constructed and delivered, did it improve what the model did?

The book now has three kinds of evidence, and this is the third.

How Much of You Can an AI Reproduce?

Thu, 01 Oct 2026 09:00:00 +0100

Priya opens a new architecture review. The system predicts that she will choose the expanded calculation before the summary. Its record says that she chose the calculation in the previous review. That record may be correct. The prediction has yet to be tested.

It is a small distinction, easily hidden by a fluent interface. “You chose this last time” becomes “you will choose this now”, and then “I know how you think”. Each sentence makes a different claim. The evidence does not travel between them merely because the system speaks with the same confidence.

Episteme: Distilling Knowledge into AI

Fri, 03 Oct 2025 11:24:58 +0100

🚀 Summary

When you can measure what you are speaking about… you know something about it; but when you cannot measure it… your knowledge is of a meagre and unsatisfactory kind. Lord Kelvin

Remember that time you spent an hour with an AI, and in one perfect response, it solved a problem you’d been stuck on for weeks? Where is that answer now? Lost in a scroll of chat history, a fleeting moment of brilliance that vanished as quickly as it appeared. This post is about how to make that moment permanent, and turn it into an intelligence that amplifies everything you do.