Autograd on AIBussin — AI applications, systems and books

What Are We Actually Doing?

Sat, 08 Aug 2026 12:40:00 +0100

It has never been easier to obtain working PyTorch code. Describe a model to an assistant, paste the result into a file, and a training loop will usually appear and usually run. For a large fraction of everyday work this is fine, and this book does not ask you to pretend otherwise.

The difficulty arrives afterwards.

The code runs, but the loss sits at exactly the same value for two hundred steps. Or one tensor is [32, 128] where the next operation wanted [128, 32]. Or the gradient of a weight is None and nothing says why. Or the model trained yesterday and today it does not, with no change you can point to. None of these are syntax errors. They do not produce a helpful traceback, and often they produce no traceback at all. They are questions about what the program is doing, and to answer them you need somewhere to look.

Autograd: What Did PyTorch Record, and Where Does the Gradient Stop?

Sat, 08 Aug 2026 12:55:00 +0100

Here is a training loop with two parameters. It runs, it does not warn, and the loss goes to zero.

import torch

x = torch.tensor([1.0, 2.0, 3.0, 4.0])
target = torch.tensor([3.0, 6.0, 9.0, 12.0])

w1 = torch.tensor(0.5, requires_grad=True)
w2 = torch.tensor(1.0, requires_grad=True)
params = [w1, w2]

feature_log = []

def stage_one(t):
 f = w1 * t
 detached = f.detach() # keep the values for later inspection
 feature_log.append(detached)
 return detached

learning_rate = 0.02

for step in range(400):
 f = stage_one(x)
 prediction = w2 * f
 loss = ((prediction - target) ** 2).mean()

 loss.backward()

 with torch.no_grad():
 for p in params:
 if p.grad is not None:
 p -= learning_rate * p.grad
 p.grad.zero_()

 if step in (0, 9, 99, 399):
 print(f"step={step:3d} loss={loss.item():.6f} "
 f"w1={w1.item():.4f} w2={w2.item():.4f}")

step= 0 loss=46.875000 w1=0.5000 w2=1.3750
step= 9 loss=11.521060 w1=0.5000 w2=3.7071
step= 99 loss=0.000009 w1=0.5000 w2=5.9979
step=399 loss=0.000000 w1=0.5000 w2=6.0000

Read the columns the way Chapter 1 taught. The loss falls to zero, which is the outcome we asked for. And w1 never moves. Not slowly, not by a little: it holds 0.5000 for four hundred steps while w2 climbs to exactly 6.0, which is the value that makes w1 * w2 equal to the 3 we were trying to learn. Half the model was frozen, the other half absorbed the entire job, and the objective was satisfied anyway.

Appendix A: PyTorch Diagnostic Field Guide

Sun, 30 Aug 2026 12:00:00 +0100

This appendix introduces no new PyTorch mechanism.

Everything here was earned earlier in the book by building something small, breaking it deliberately, measuring what changed, and locating the first place where reality stopped matching the intended computation.

The purpose of this appendix is different.

When a real model is failing, you usually do not need another explanation of autograd, broadcasting, attention or CUDA. You need to answer a narrower question:

PyTorch Autograd Debugging: requires_grad, detach, backward() and NaN Gradients

Sat, 08 Aug 2026 12:55:00 +0100

PyTorch: Zero to Hero — Step 02

In the previous post we treated tensor shapes as a debugging problem rather than a mathematical vocabulary exercise.

We are going to do the same thing with autograd.

If you use PyTorch for any serious amount of time, you eventually see errors like:

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

or:

RuntimeError: Trying to backward through the graph a second time...

or worse: