Guide · AI infrastructure

How to keep GPUs fed.

Where GPU time quietly leaks to data movement — staging datasets, checkpointing, and moving data between sites and clouds — what it costs, and a practical playbook for keeping an AI factory supplied at line rate.

Updated November 18, 2026 · by the Zettar team
The short answer. GPUs bill by the hour whether they compute or wait. Outside the training loop, they wait on three data-movement jobs: staging datasets into the cluster, writing and restoring checkpoints, and moving data and weights between sites and clouds. Keep GPUs fed by staging data ahead of the job, putting a fast NVMe tier between storage and the network, moving data in parallel at line rate, and offloading checkpoints without stalling training. At line rate, a 100 Gbps link stages a petabyte in about a day; at 20% utilization, the same petabyte takes nearly five.

Where does GPU time leak to data?

  • Staging training data. Datasets arrive from object storage, other sites, partners, or the cloud, and must land in the cluster’s storage before a job can start. A slow transfer delays every run that depends on it.
  • Checkpointing. Large models write checkpoints — which can reach terabytes once optimizer state is included — on a regular schedule, and must read them back after a failure. Copies also go off-cluster for durability, evaluation, or another site.
  • Moving between sites and clouds. Multi-site AI factories, bursting to rented GPUs, sovereign deployments, and model distribution all move datasets and weights across long, fast paths.
  • Feeding the job itself. Inside a running job, data loaders, preprocessing, and the cluster file system have to keep up. That is tuned within the cluster — but it only works if the data is already there, on fast storage close to the GPUs.

What does waiting cost?

Idle GPUs still bill. A 2,048-GPU cluster at $3 per GPU-hour that waits on data 15% of the time loses about $8M a year in GPU time alone — the AI defaults in our cost-of-slow-data calculator, before counting the engineers who wait too. Plug in your own numbers.

How long does staging really take?

Throughput, not link speed, sets staging time. The difference between a tool that realizes a fraction of the link and a path running at line rate is days per petabyte.

Data to stage100 Gbps at 20%100 Gbps at line rate400 Gbps at line rate
100 TB11 h2.5 h37 min
1 PB4.6 days25 h6.2 h
5 PB23 days5.1 days31 h

Line rate = about 90% of the link. Formula: hours = data (TB) ÷ (link Gbps × utilization × 0.45 TB per hour per Gbps). See how to transfer petabytes of data.

A playbook for keeping GPUs fed

  • Stage ahead and overlap. Move the next dataset while the current job trains, so staging never sits on the critical path.
  • Put fast storage next to the network. An NVMe burst buffer absorbs bursty storage I/O so transfers run at the speed of the wire, not the slowest disk.
  • Move in parallel, at line rate. Many streams across many nodes, scaling out with the link — no single connection or server should cap throughput.
  • Offload checkpoints without stalling training. Write checkpoints to a fast tier, then copy them off-cluster and to other sites in the background.
  • Keep native formats. Move files and objects as they are, so data lands ready for the loader with nothing to repack.
  • Verify every byte. End-to-end checksums catch silent corruption before it reaches a training run.
  • Engineer the cloud path. Standard cloud transfer paths carry a performance penalty; a co-designed path avoids most of it.

Questions to ask about your pipeline

  • How long does it take today to stage a full training dataset — and how often do jobs wait for it?
  • How long do checkpoint writes, restores, and off-cluster copies take, and do they pause training?
  • What share of your inter-site and cloud links do transfers actually use?
  • Where is the bottleneck — storage, hosts, software, or the network? (Usually not the network.)

What line rate looks like in practice

With SLAC National Accelerator Laboratory and ESnet (U.S. DOE), Zettar moved 1 PB in 29 hours over a 5,000-mile path at 96% link utilization, encrypted and checksummed — a petabyte a day on a full 100 Gbps link. Moving a 1.2 TiB dataset from KEK in Japan to AWS N. Virginia, zx finished in about 40 minutes versus 235 minutes for aws-cli (Fang et al., arXiv:2512.15028).

The Zettar zx Appliance is the data-movement supply line for an AI factory: storage, host, network, and software co-designed, with an NVMe burst buffer, delivered turnkey and pretuned. See AI datacenters.

What do idle GPUs cost you? → Estimate a staging time →

FAQ

Common questions

Why do GPUs sit idle waiting on data?

Outside the training loop, GPUs wait on data movement: staging datasets into the cluster, writing and restoring checkpoints, and moving data and weights between sites and clouds. When those transfers realize only a fraction of the link, jobs start late and stall, while the GPUs keep billing.

How long does it take to stage a petabyte of training data?

At line rate on a 100 Gbps link (about 90% utilization), roughly a day; on a 400 Gbps link, about six hours. At 20% utilization, the same petabyte takes nearly five days on 100 Gbps.

How do you keep checkpointing from stalling training?

Write checkpoints to a fast NVMe tier, then copy them off-cluster and to other sites in the background at line rate, so the job is not waiting on slow storage or a slow transfer.

Is data movement or compute the real bottleneck for AI?

As clusters grow, data movement increasingly is. Compute scales by adding GPUs, but every GPU still waits if datasets, checkpoints, and weights cannot reach it fast enough — and the bottleneck is usually storage, hosts, and software rather than the network.

How does Zettar help keep GPUs fed?

The Zettar zx Appliance is a co-designed data-movement layer — storage, host, network, and software engineered together with an NVMe burst buffer — that stages datasets, offloads checkpoints, and moves data between sites and clouds at line rate, about 90% or more of available bandwidth.

See it for yourself

Stop paying for idle GPUs.

See the appliance stage your data at line rate — or talk through your pipeline with an engineer.