Language Models on AIBussin — AI applications, systems and books

When Models Make Things Up

Sat, 29 Aug 2026 22:10:00 +0100

A language model can produce an answer that is fluent, grammatical, detailed, confident, and wrong.

That combination is what makes hallucination interesting.

Ordinary software failures usually leave evidence. A parser throws an exception. A database rejects an invalid constraint. A test fails. A network call times out. The failure changes the shape of the result.

A hallucinating language model can do something more difficult to handle: it can fail without looking broken.

Your Model Is a Dependency

Fri, 28 Aug 2026 10:25:00 +0100

Chapter 5 built a composed program and measured which of its stages earned their cost. Every number in that chapter, and in chapter 4 before it, has an unstated qualifier:

Which model executed the program?

That is not operational trivia. A language-model program is not fully described by its signatures and modules. Its behavior depends on what sits behind the LM boundary and how that thing is configured.

The model is therefore both a dependency and an experimental variable. Holding it fixed is often as important as swapping it:

Assembly: A Language Model You Can Interrogate

Sun, 30 Aug 2026 10:00:00 +0100

Here are two training runs of the same small language model on the same data. The only difference is one line in the batching function.

CORRECT y = data[i+1 : i+block+1] the next-token shift
MISALIGNED y = data[i : i+block] no shift

step correct val misaligned val
 0 3.6695 2.6469
 250 1.1038 0.0007
 500 0.4730 0.0004
 750 0.3940 0.0003
1000 0.3690 0.0002
1250 0.3581 0.0002
1500 0.3558 0.0001

The misaligned run’s loss falls to 0.0001 — three thousand times lower than the correct run’s 0.36. By every number on the dashboard it is the best training run you have ever seen. Nothing raised. The shapes are identical: x is [B, T], y is [B, T], the loss is a finite scalar with a grad_fn.

PyTorch Zero to Hero 10: Build a Small GPT-Style Language Model From Scratch

Sat, 08 Aug 2026 14:03:00 +0100

Build a Small GPT-Style Language Model From Scratch in PyTorch

This is the final post in the PyTorch: Zero to Hero series.

We have spent the previous posts learning the machinery underneath PyTorch:

tensors and shapes;
autograd;
manual neural networks;
nn.Module and parameter registration;
DataLoader performance;
convolutional networks;
attention and masks;
training failures;
CUDA performance and torch.compile.

Now we put it together.

The goal is not to download a pretrained model.