DataLoader: Where Is the Training Loop Actually Waiting?

Sat, 08 Aug 2026 13:21:00 +0100

Here are two controlled training pipelines. They use the same batch size, the same machine, and the same synthetic post-batch workload. Each sample also carries the same nominal two-millisecond cost: in one pipeline that cost is waiting, while in the other it is fixed CPU work.

The tensor construction around that controlled cost is the same in both cases. What changes is the resource those two milliseconds consume.

Both are given the same treatment — raise num_workers from 0 to 8 — and measured the same way.

PyTorch DataLoader Performance: num_workers, pin_memory, Prefetching and Why Your GPU Is Waiting

Sat, 08 Aug 2026 13:21:00 +0100

PyTorch: Zero to Hero — Step 05

A fast model with a slow input pipeline is still a slow training system.

One of the most common PyTorch performance failures looks like this:

GPU utilization: 20% → 95% → 10% → 90% → 15%

The model is not necessarily slow.

The GPU may simply be waiting for the next batch.

This article is about finding out where the wait is happening.

Pin_memory on AIBussin — AI applications, systems and books

DataLoader: Where Is the Training Loop Actually Waiting?

PyTorch DataLoader Performance: num_workers, pin_memory, Prefetching and Why Your GPU Is Waiting

PyTorch: Zero to Hero — Step 05