Transformers on AIBussin — AI applications, systems and books

Attention: Which Position Is Comparing With Which?

Sat, 29 Aug 2026 10:00:00 +0100

Here is a tensor of sequence representations and the line that splits it into heads.

B, T, E, Nh = 1, 4, 8, 2
Dh = E // Nh # 4

x = torch.arange(B * T * E).reshape(B, T, E)
heads = x.reshape(B, Nh, T, Dh)

x: (1, 4, 8)
heads: (1, 2, 4, 4)

That is exactly the shape multi-head attention wants: batch, heads, positions, head dimension. Nothing raised. Every shape assertion passes.

Now the same split written the other way:

Assembly: A Language Model You Can Interrogate

Sun, 30 Aug 2026 10:00:00 +0100

Here are two training runs of the same small language model on the same data. The only difference is one line in the batching function.

CORRECT y = data[i+1 : i+block+1] the next-token shift
MISALIGNED y = data[i : i+block] no shift

step correct val misaligned val
 0 3.6695 2.6469
 250 1.1038 0.0007
 500 0.4730 0.0004
 750 0.3940 0.0003
1000 0.3690 0.0002
1250 0.3581 0.0002
1500 0.3558 0.0001

The misaligned run’s loss falls to 0.0001 — three thousand times lower than the correct run’s 0.36. By every number on the dashboard it is the best training run you have ever seen. Nothing raised. The shapes are identical: x is [B, T], y is [B, T], the loss is a finite scalar with a grad_fn.

PyTorch Zero to Hero 10: Build a Small GPT-Style Language Model From Scratch

Sat, 08 Aug 2026 14:03:00 +0100

Build a Small GPT-Style Language Model From Scratch in PyTorch

This is the final post in the PyTorch: Zero to Hero series.

We have spent the previous posts learning the machinery underneath PyTorch:

tensors and shapes;
autograd;
manual neural networks;
nn.Module and parameter registration;
DataLoader performance;
convolutional networks;
attention and masks;
training failures;
CUDA performance and torch.compile.

Now we put it together.

The goal is not to download a pretrained model.

PyTorch Attention Shapes: Q, K, V, Multi-Head Attention Masks and Transformer Dimension Errors

Sat, 08 Aug 2026 13:30:00 +0100

PyTorch: Zero to Hero — Step 07

Attention code is where tensor-shape mistakes stop being annoying and start becoming architectural.

A CNN usually makes its dimensional assumptions fairly obvious. Attention does not.

A tensor that starts as:

(batch, sequence, embedding)

is projected into Q, K and V, split into heads, transposed, multiplied, masked, normalized, multiplied again, transposed again, concatenated and projected back to the embedding dimension.

A single bad view, transpose, mask shape or head calculation can produce anything from an immediate runtime error to a model that trains while attending to the wrong tokens.