Performance on AIBussin — AI applications, systems and books

DataLoader: Where Is the Training Loop Actually Waiting?

Sat, 08 Aug 2026 13:21:00 +0100

Here are two controlled training pipelines. They use the same batch size, the same machine, and the same synthetic post-batch workload. Each sample also carries the same nominal two-millisecond cost: in one pipeline that cost is waiting, while in the other it is fixed CPU work.

The tensor construction around that controlled cost is the same in both cases. What changes is the resource those two milliseconds consume.

Both are given the same treatment — raise num_workers from 0 to 8 — and measured the same way.

Cold Starts, Warm Runs and Real Latency

Wed, 02 Sep 2026 15:30:00 +0100

“Local AI is faster” is not a measurement.

It compresses several different waiting periods into one adjective. A browser-native feature can avoid network round trips and still make a user wait for model acquisition, process startup, session creation, context ingestion or slow generation.

To understand latency, we have to take the lifecycle apart.

1. One duration hides several clocks

For one interaction, useful timestamps include:

Phase	Starts	Ends	User-visible?
Capability inspection	`availability()` call	state returned	Usually not
Acquisition	session request requires assets	assets ready	Yes, if it blocks
Session creation	`create()` called	session returned	Often
Prompt startup	prompt called	first chunk	Yes
Generation	first chunk	final chunk	Yes
Validation	output complete	feature accepted/rejected	Sometimes

The perceived wait for a streaming feature is often dominated by time to first output:

Performance: What Is the Machine Waiting For?

Sat, 29 Aug 2026 16:00:00 +0100

Here is a benchmark. A model, a batch, a loop, a timer. Nothing exotic.

model.train()
t0 = time.perf_counter()
for _ in range(20):
 optimizer.zero_grad(set_to_none=True)
 loss = F.cross_entropy(model(x), y) # MLP, batch 256
 loss.backward()
 optimizer.step()
ms_per_step = (time.perf_counter() - t0) / 20 * 1000

naive loop of 20: 9.491 ms/step reported + 132.0 ms still queued at the final sync -> really 16.093 ms/step
naive loop of 100: 14.692 ms/step reported + 138.8 ms still queued at the final sync -> really 16.080 ms/step
naive loop of 400: 15.741 ms/step reported + 132.1 ms still queued at the final sync -> really 16.071 ms/step

synchronized + warmed, median of 400 : 16.430 ms/step <- the real number
one step, host-visible return only : 4.866 ms/step
first training step ever : 20.890 ms

Read the first line again. The loop ran twenty iterations and reported 9.491 ms per step. Then a single torch.cuda.synchronize() immediately afterward blocked for 132 more milliseconds — roughly eight more steps’ worth of work that the GPU had not finished when the timer stopped. Loop over a hundred iterations instead and the same code reports 14.7. Over four hundred, 15.7. Time one step in isolation and it looks like 4.9.

Appendix A: PyTorch Diagnostic Field Guide

Sun, 30 Aug 2026 12:00:00 +0100

This appendix introduces no new PyTorch mechanism.

Everything here was earned earlier in the book by building something small, breaking it deliberately, measuring what changed, and locating the first place where reality stopped matching the intended computation.

The purpose of this appendix is different.

When a real model is failing, you usually do not need another explanation of autograd, broadcasting, attention or CUDA. You need to answer a narrower question:

Profile Before You Optimize

Mon, 10 Aug 2026 20:52:00 +0100

By now we have built dozens of automata.

The natural temptation is to make them faster.

The wrong first question is:

Which optimization trick should I use?

The right first question is:

Where is the time actually going?

Performance work begins with measurement, and with the doctrine Part VI starts from:

Optimize the verified implementation, not the first implementation that happens to render something plausible.

Running this book’s own earlier code turned up undefined helpers, wrong signatures and a misaligned FFT, all inside chapters that produced plausible figures. Speed means nothing until equivalence is established.

PyTorch Performance Debugging: CUDA OOM, Slow Training, GPU Utilization and torch.compile

Sat, 08 Aug 2026 13:56:00 +0100

PyTorch: Zero to Hero — Step 09

At this point in the series, the model runs.

That does not mean it runs well.

A training loop can be correct and still waste most of the machine.

A model can fit in memory and still spend half its time waiting on synchronization.

A torch.compile call can make code faster, slower, or simply move the bottleneck somewhere else.

A CUDA out-of-memory error can be caused by the model, the optimizer, activations, fragmentation, a leaked reference, a larger batch, a longer sequence, or an innocent-looking tensor that was kept alive by Python.

PyTorch DataLoader Performance: num_workers, pin_memory, Prefetching and Why Your GPU Is Waiting

Sat, 08 Aug 2026 13:21:00 +0100

PyTorch: Zero to Hero — Step 05

A fast model with a slow input pipeline is still a slow training system.

One of the most common PyTorch performance failures looks like this:

GPU utilization: 20% → 95% → 10% → 90% → 15%

The model is not necessarily slow.

The GPU may simply be waiting for the next batch.

This article is about finding out where the wait is happening.