Attention on AIBussin — AI applications, systems and books

Make the Window Bigger

Wed, 23 Sep 2026 05:00:08 +0100

Seven chapters have treated the model as fixed ground and asked how the surrounding system should manage information against it. The natural objection has been waiting since Chapter 4: if context is scarce and difficult to manage, why not simply make the window much larger? A team that moves its agent onto a million-token model watches several selection problems relax at once. Histories that needed pruning now fit. Files that needed triage can all be included. The compaction schedule gets quieter. For a while, context engineering looks like a stopgap awaiting cheaper capacity.

Designing Your Digital Lens

Mon, 01 Jan 0001 00:00:00 +0000

AI as an Attention Interface

The previous chapter looked inward: reflection, routines, personal notes, and companion roles. This chapter looks outward.

Most of digital life is not scarce. It is excessive. Feeds update constantly. Notifications arrive from systems with different incentives. Search returns more than you can read. News, entertainment, work, advertising, research, and social pressure all arrive through the same devices.

A digital lens is an agentic layer that helps filter, prioritize, summarize, or reshape that flow according to a purpose you choose.

Attention: Which Position Is Comparing With Which?

Sat, 29 Aug 2026 10:00:00 +0100

Here is a tensor of sequence representations and the line that splits it into heads.

B, T, E, Nh = 1, 4, 8, 2
Dh = E // Nh # 4

x = torch.arange(B * T * E).reshape(B, T, E)
heads = x.reshape(B, Nh, T, Dh)

x: (1, 4, 8)
heads: (1, 2, 4, 4)

That is exactly the shape multi-head attention wants: batch, heads, positions, head dimension. Nothing raised. Every shape assertion passes.

Now the same split written the other way:

Too Much Information

Fri, 25 Sep 2026 02:45:00 +0100

The Semantic Browser works. On any page you open, it chooses a representation, enriches it, types its relationships, and gates everything through preservation evidence. Now count what it never touches: the forty hours of conference video in your subscriptions, the papers published this week in your field, the podcasts queued behind the podcasts, the threads, the advisories, the second-order citations of everything you read. The browser perfects the encounter with information already in front of you. It does nothing for the information you never reach — which is nearly all of it.

Assembly: A Language Model You Can Interrogate

Sun, 30 Aug 2026 10:00:00 +0100

Here are two training runs of the same small language model on the same data. The only difference is one line in the batching function.

CORRECT y = data[i+1 : i+block+1] the next-token shift
MISALIGNED y = data[i : i+block] no shift

step correct val misaligned val
 0 3.6695 2.6469
 250 1.1038 0.0007
 500 0.4730 0.0004
 750 0.3940 0.0003
1000 0.3690 0.0002
1250 0.3581 0.0002
1500 0.3558 0.0001

The misaligned run’s loss falls to 0.0001 — three thousand times lower than the correct run’s 0.36. By every number on the dashboard it is the best training run you have ever seen. Nothing raised. The shapes are identical: x is [B, T], y is [B, T], the loss is a finite scalar with a grad_fn.

Appendix A: PyTorch Diagnostic Field Guide

Sun, 30 Aug 2026 12:00:00 +0100

This appendix introduces no new PyTorch mechanism.

Everything here was earned earlier in the book by building something small, breaking it deliberately, measuring what changed, and locating the first place where reality stopped matching the intended computation.

The purpose of this appendix is different.

When a real model is failing, you usually do not need another explanation of autograd, broadcasting, attention or CUDA. You need to answer a narrower question:

Inside Tiny — Residual Blocks, Attention and Sparse Autoencoders

Sat, 08 Aug 2026 15:00:00 +0100

Inside Tiny: Residual Blocks, Attention and Sparse Autoencoders

In the previous post, we built a compact recursive model around one idea:

context + candidate + latent state
 ↓
 projection
 ↓
 reusable core
 ↓
 proposed update
 ↓
 z ← z + α · update
 ↓
 repeat

That architecture looked more sophisticated than MR.Q, EBT or SICQL because it introduced recurrence.

But the central idea of this series is that a model stops looking mysterious when we keep opening it.

PyTorch Attention Shapes: Q, K, V, Multi-Head Attention Masks and Transformer Dimension Errors

Sat, 08 Aug 2026 13:30:00 +0100

PyTorch: Zero to Hero — Step 07

Attention code is where tensor-shape mistakes stop being annoying and start becoming architectural.

A CNN usually makes its dimensional assumptions fairly obvious. Attention does not.

A tensor that starts as:

(batch, sequence, embedding)

is projected into Q, K and V, split into heads, transposed, multiplied, masked, normalized, multiplied again, transposed again, concatenated and projected back to the embedding dimension.

A single bad view, transpose, mask shape or head calculation can produce anything from an immediate runtime error to a model that trains while attending to the wrong tokens.