Where does GPU time leak to data?
- Staging training data. Datasets arrive from object storage, other sites, partners, or the cloud, and must land in the cluster’s storage before a job can start. A slow transfer delays every run that depends on it.
- Checkpointing. Large models write checkpoints — which can reach terabytes once optimizer state is included — on a regular schedule, and must read them back after a failure. Copies also go off-cluster for durability, evaluation, or another site.
- Moving between sites and clouds. Multi-site AI factories, bursting to rented GPUs, sovereign deployments, and model distribution all move datasets and weights across long, fast paths.
- Feeding the job itself. Inside a running job, data loaders, preprocessing, and the cluster file system have to keep up. That is tuned within the cluster — but it only works if the data is already there, on fast storage close to the GPUs.
What does waiting cost?
How long does staging really take?
Throughput, not link speed, sets staging time. The difference between a tool that realizes a fraction of the link and a path running at line rate is days per petabyte.
| Data to stage | 100 Gbps at 20% | 100 Gbps at line rate | 400 Gbps at line rate |
|---|---|---|---|
| 100 TB | 11 h | 2.5 h | 37 min |
| 1 PB | 4.6 days | 25 h | 6.2 h |
| 5 PB | 23 days | 5.1 days | 31 h |
Line rate = about 90% of the link. Formula: hours = data (TB) ÷ (link Gbps × utilization × 0.45 TB per hour per Gbps). See how to transfer petabytes of data.
A playbook for keeping GPUs fed
- Stage ahead and overlap. Move the next dataset while the current job trains, so staging never sits on the critical path.
- Put fast storage next to the network. An NVMe burst buffer absorbs bursty storage I/O so transfers run at the speed of the wire, not the slowest disk.
- Move in parallel, at line rate. Many streams across many nodes, scaling out with the link — no single connection or server should cap throughput.
- Offload checkpoints without stalling training. Write checkpoints to a fast tier, then copy them off-cluster and to other sites in the background.
- Keep native formats. Move files and objects as they are, so data lands ready for the loader with nothing to repack.
- Verify every byte. End-to-end checksums catch silent corruption before it reaches a training run.
- Engineer the cloud path. Standard cloud transfer paths carry a performance penalty; a co-designed path avoids most of it.
Questions to ask about your pipeline
- How long does it take today to stage a full training dataset — and how often do jobs wait for it?
- How long do checkpoint writes, restores, and off-cluster copies take, and do they pause training?
- What share of your inter-site and cloud links do transfers actually use?
- Where is the bottleneck — storage, hosts, software, or the network? (Usually not the network.)
What line rate looks like in practice
With SLAC National Accelerator Laboratory and ESnet (U.S. DOE), Zettar moved 1 PB in 29 hours over a 5,000-mile path at 96% link utilization, encrypted and checksummed — a petabyte a day on a full 100 Gbps link. Moving a 1.2 TiB dataset from KEK in Japan to AWS N. Virginia, zx finished in about 40 minutes versus 235 minutes for aws-cli (Fang et al., arXiv:2512.15028).
The Zettar zx Appliance is the data-movement supply line for an AI factory: storage, host, network, and software co-designed, with an NVMe burst buffer, delivered turnkey and pretuned. See AI datacenters.
