Backpropagation on AIBussin — AI applications, systems and books

Autograd: What Did PyTorch Record, and Where Does the Gradient Stop?

Sat, 08 Aug 2026 12:55:00 +0100

Here is a training loop with two parameters. It runs, it does not warn, and the loss goes to zero.

import torch

x = torch.tensor([1.0, 2.0, 3.0, 4.0])
target = torch.tensor([3.0, 6.0, 9.0, 12.0])

w1 = torch.tensor(0.5, requires_grad=True)
w2 = torch.tensor(1.0, requires_grad=True)
params = [w1, w2]

feature_log = []

def stage_one(t):
 f = w1 * t
 detached = f.detach() # keep the values for later inspection
 feature_log.append(detached)
 return detached

learning_rate = 0.02

for step in range(400):
 f = stage_one(x)
 prediction = w2 * f
 loss = ((prediction - target) ** 2).mean()

 loss.backward()

 with torch.no_grad():
 for p in params:
 if p.grad is not None:
 p -= learning_rate * p.grad
 p.grad.zero_()

 if step in (0, 9, 99, 399):
 print(f"step={step:3d} loss={loss.item():.6f} "
 f"w1={w1.item():.4f} w2={w2.item():.4f}")

step= 0 loss=46.875000 w1=0.5000 w2=1.3750
step= 9 loss=11.521060 w1=0.5000 w2=3.7071
step= 99 loss=0.000009 w1=0.5000 w2=5.9979
step=399 loss=0.000000 w1=0.5000 w2=6.0000

Read the columns the way Chapter 1 taught. The loss falls to zero, which is the outcome we asked for. And w1 never moves. Not slowly, not by a little: it holds 0.5000 for four hundred steps while w2 climbs to exactly 6.0, which is the value that makes w1 * w2 equal to the 3 we were trying to learn. Half the model was frozen, the other half absorbed the entire job, and the objective was satisfied anyway.

The Network: What Is It Without nn.Module?

Sat, 08 Aug 2026 12:59:00 +0100

Here is a two-layer neural network, written with nothing but the machinery of the last three chapters. Four trainable tensors, a forward pass, a loss, backward(), an update. It runs without warnings, and the loss falls by a third.

import math
import torch

torch.manual_seed(0)

n = 2000
X = torch.randn(n, 2)
y = ((X[:, 0] > 0) ^ (X[:, 1] > 0)).long()

perm = torch.randperm(n)
n_train = int(0.8 * n)
X_train, y_train = X[perm[:n_train]], y[perm[:n_train]]
X_val, y_val = X[perm[n_train:]], y[perm[n_train:]]

D_in, H, C = 2, 16, 2

torch.manual_seed(1)
W1 = (torch.randn(D_in, H) / math.sqrt(D_in)).requires_grad_()
b1 = torch.zeros(H, requires_grad=True)
W2 = (torch.randn(H, C) / math.sqrt(H)).requires_grad_()
b2 = torch.zeros(C, requires_grad=True)
parameters = [W1, b1, W2, b2]

def forward(x):
 z1 = x @ W1 + b1
 return z1 @ W2 + b2

def cross_entropy(logits, targets):
 log_probs = torch.log_softmax(logits, dim=1)
 return -log_probs[torch.arange(targets.shape[0]), targets].mean()

def accuracy(logits, targets):
 return (logits.argmax(dim=1) == targets).float().mean().item()

learning_rate = 0.5

# First verify that one training step is mechanically healthy.
names = ["W1", "b1", "W2", "b2"]
initial_values = [p.detach().clone() for p in parameters]

loss = cross_entropy(forward(X_train), y_train)
before = [p.detach().clone() for p in parameters]

loss.backward()

with torch.no_grad():
 for p in parameters:
 p -= learning_rate * p.grad

for name, p, old in zip(names, parameters, before):
 print(f"{name:3s} leaf={p.is_leaf} grad_none={p.grad is None} "
 f"grad_norm={p.grad.norm().item():8.4f} "
 f"finite={torch.isfinite(p.grad).all().item()} "
 f"moved={(p.detach() - old).abs().max().item():.4f}")

W1 leaf=True grad_none=False grad_norm= 0.4516 finite=True moved=0.1048
b1 leaf=True grad_none=False grad_norm= 0.0192 finite=True moved=0.0053
W2 leaf=True grad_none=False grad_norm= 1.1058 finite=True moved=0.1918
b2 leaf=True grad_none=False grad_norm= 0.0175 finite=True moved=0.0062

Every parameter is a leaf. Every parameter requires gradients. Every parameter receives a finite gradient, and every parameter changes by a measurable amount when the update runs. The basic training machinery is working.

Build a Neural Network From Scratch in PyTorch Without nn.Module

Sat, 08 Aug 2026 12:59:00 +0100

PyTorch: Zero to Hero — Step 03

Most PyTorch tutorials begin with something like this:

import torch.nn as nn

model = nn.Sequential(
 nn.Linear(2, 16),
 nn.ReLU(),
 nn.Linear(16, 2),
)

That is useful code.

It is also hiding almost everything interesting.

In this post we are going to build the same kind of neural network using ordinary PyTorch tensors.

No nn.Module.

No nn.Linear.

No torch.optim.Adam.

No optimizer.step().

We will manually create the parameters, write the forward pass, calculate the loss, call autograd, update the weights, zero the gradients, batch the data, evaluate the model and then compare the result with the idiomatic PyTorch version.

PyTorch Autograd Debugging: requires_grad, detach, backward() and NaN Gradients

Sat, 08 Aug 2026 12:55:00 +0100

PyTorch: Zero to Hero — Step 02

In the previous post we treated tensor shapes as a debugging problem rather than a mathematical vocabulary exercise.

We are going to do the same thing with autograd.

If you use PyTorch for any serious amount of time, you eventually see errors like:

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

or:

RuntimeError: Trying to backward through the graph a second time...

or worse: