Torch.compile on AIBussin — AI applications, systems and books

Performance: What Is the Machine Waiting For?

Sat, 29 Aug 2026 16:00:00 +0100

Here is a benchmark. A model, a batch, a loop, a timer. Nothing exotic.

model.train()
t0 = time.perf_counter()
for _ in range(20):
 optimizer.zero_grad(set_to_none=True)
 loss = F.cross_entropy(model(x), y) # MLP, batch 256
 loss.backward()
 optimizer.step()
ms_per_step = (time.perf_counter() - t0) / 20 * 1000

naive loop of 20: 9.491 ms/step reported + 132.0 ms still queued at the final sync -> really 16.093 ms/step
naive loop of 100: 14.692 ms/step reported + 138.8 ms still queued at the final sync -> really 16.080 ms/step
naive loop of 400: 15.741 ms/step reported + 132.1 ms still queued at the final sync -> really 16.071 ms/step

synchronized + warmed, median of 400 : 16.430 ms/step <- the real number
one step, host-visible return only : 4.866 ms/step
first training step ever : 20.890 ms

Read the first line again. The loop ran twenty iterations and reported 9.491 ms per step. Then a single torch.cuda.synchronize() immediately afterward blocked for 132 more milliseconds — roughly eight more steps’ worth of work that the GPU had not finished when the timer stopped. Loop over a hundred iterations instead and the same code reports 14.7. Over four hundred, 15.7. Time one step in isolation and it looks like 4.9.

Compilation: Which Assumption Stopped Holding?

Sat, 29 Aug 2026 17:00:00 +0100

Here is an inference service. It loads a model, compiles it once, and serves requests whose sequence length varies from request to request — the ordinary situation for anything that takes text.

model = load_model().eval()
compiled = torch.compile(model, dynamic=False) # "dynamic shapes are slow, force static"

for request in stream:
 with torch.no_grad():
 output = compiled(request.tokens) # [1, T], T varies per request

Thirteen requests. Every compiled output agrees with the eager model to a maximum absolute difference of 7e-7. Nothing raises.

PyTorch Performance Debugging: CUDA OOM, Slow Training, GPU Utilization and torch.compile

Sat, 08 Aug 2026 13:56:00 +0100

PyTorch: Zero to Hero — Step 09

At this point in the series, the model runs.

That does not mean it runs well.

A training loop can be correct and still waste most of the machine.

A model can fit in memory and still spend half its time waiting on synchronization.

A torch.compile call can make code faster, slower, or simply move the bottleneck somewhere else.

A CUDA out-of-memory error can be caused by the model, the optimizer, activations, fragmentation, a leaked reference, a larger batch, a longer sequence, or an innocent-looking tensor that was kept alive by Python.