Gradients on AIBussin — AI applications, systems and books

What Are We Actually Doing?

Sat, 08 Aug 2026 12:40:00 +0100

It has never been easier to obtain working PyTorch code. Describe a model to an assistant, paste the result into a file, and a training loop will usually appear and usually run. For a large fraction of everyday work this is fine, and this book does not ask you to pretend otherwise.

The difficulty arrives afterwards.

The code runs, but the loss sits at exactly the same value for two hundred steps. Or one tensor is [32, 128] where the next operation wanted [128, 32]. Or the gradient of a weight is None and nothing says why. Or the model trained yesterday and today it does not, with no change you can point to. None of these are syntax errors. They do not produce a helpful traceback, and often they produce no traceback at all. They are questions about what the program is doing, and to answer them you need somewhere to look.

Autograd: What Did PyTorch Record, and Where Does the Gradient Stop?

Sat, 08 Aug 2026 12:55:00 +0100

Here is a training loop with two parameters. It runs, it does not warn, and the loss goes to zero.

import torch

x = torch.tensor([1.0, 2.0, 3.0, 4.0])
target = torch.tensor([3.0, 6.0, 9.0, 12.0])

w1 = torch.tensor(0.5, requires_grad=True)
w2 = torch.tensor(1.0, requires_grad=True)
params = [w1, w2]

feature_log = []

def stage_one(t):
 f = w1 * t
 detached = f.detach() # keep the values for later inspection
 feature_log.append(detached)
 return detached

learning_rate = 0.02

for step in range(400):
 f = stage_one(x)
 prediction = w2 * f
 loss = ((prediction - target) ** 2).mean()

 loss.backward()

 with torch.no_grad():
 for p in params:
 if p.grad is not None:
 p -= learning_rate * p.grad
 p.grad.zero_()

 if step in (0, 9, 99, 399):
 print(f"step={step:3d} loss={loss.item():.6f} "
 f"w1={w1.item():.4f} w2={w2.item():.4f}")

step= 0 loss=46.875000 w1=0.5000 w2=1.3750
step= 9 loss=11.521060 w1=0.5000 w2=3.7071
step= 99 loss=0.000009 w1=0.5000 w2=5.9979
step=399 loss=0.000000 w1=0.5000 w2=6.0000

Read the columns the way Chapter 1 taught. The loss falls to zero, which is the outcome we asked for. And w1 never moves. Not slowly, not by a little: it holds 0.5000 for four hundred steps while w2 climbs to exactly 6.0, which is the value that makes w1 * w2 equal to the 3 we were trying to learn. Half the model was frozen, the other half absorbed the entire job, and the objective was satisfied anyway.

Training: Which Link in the Learning Chain Is Broken?

Sat, 29 Aug 2026 14:00:00 +0100

Here is an ordinary piece of transfer-learning code. A small model, a frozen feature extractor, an optimizer, and a new classification head for the new task.

model = TinyClassifier()

for p in model.features.parameters():
 p.requires_grad_(False)

optimizer = torch.optim.AdamW(model.parameters(), lr=1e-2, weight_decay=0.0)

model.head = nn.Linear(32, 2) # adaptation, performed later

Train it for two hundred steps on a balanced, perfectly aligned, entirely learnable binary task:

step= 0 loss=0.683606 acc=0.7144
step= 50 loss=0.683606 acc=0.7144
step= 100 loss=0.683606 acc=0.7144
step= 150 loss=0.683606 acc=0.7144
step= 200 loss=0.683606 acc=0.7144

Not approximately constant. Identical to six decimal places, two hundred times.

PACS — Building an Optimizer From Gradient Statistics

Sat, 08 Aug 2026 15:05:00 +0100

PACS — Building an Optimizer From Gradient Statistics

So far in Models From First Principles, every post has asked some version of the same question:

What should the model compute?

MR.Q gave us a scalar quality estimate.

EBT split one shared representation into Q, V and Policy.

SICQL made those heads explicit, replaceable components.

HRM introduced repeated computation over fast and slow latent states.

Tiny compressed iterative refinement into one recursive latent state.

PyTorch Model Not Learning? A Systematic Debugging Guide

Sat, 08 Aug 2026 13:51:00 +0100

PyTorch: Zero to Hero — Step 08

Your model runs.

The loss is finite.

Nothing crashes.

And it still does not learn.

This is one of the most frustrating states in machine learning because there is no stack trace telling you what is wrong.

The program is valid Python.

The tensors have legal shapes.

The GPU is busy.

The optimizer is stepping.

And the model is useless.

This post is a systematic way to debug that situation.

PyTorch Autograd Debugging: requires_grad, detach, backward() and NaN Gradients

Sat, 08 Aug 2026 12:55:00 +0100

PyTorch: Zero to Hero — Step 02

In the previous post we treated tensor shapes as a debugging problem rather than a mathematical vocabulary exercise.

We are going to do the same thing with autograd.

If you use PyTorch for any serious amount of time, you eventually see errors like:

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

or:

RuntimeError: Trying to backward through the graph a second time...

or worse: