Training vs Inference: Two Different Infra Problems

The industry keeps building one kind of cluster and acting like it's both. The hardware overlaps, but training and inference stress completely different parts of the system. If you size inference like training, you overpay for fabric. If you size training like inference, your all-reduce falls apart.

TL;DR

• Training = one giant synchronous job. You care about scale-out bandwidth, collective efficiency, failure recovery, and power stability.
• Inference = millions of small, independent, latency-sensitive jobs. Decode is memory-bandwidth-bound, and capacity is gated by KV cache.
• Llama 3 70B needs ~320 KB of KV cache per token. One 128K-token sequence is ~40 GB. That's why disaggregated prefill/decode and tiered KV caches are now the default architecture.

Training: the cluster is the computer

A pretraining job runs thousands of GPUs in lockstep. Every step does compute, then collective communication (all-reduce for data parallel, all-to-all for MoE experts), then repeats. That gives you some specific infra requirements:

• Non-blocking scale-out fabric. The slowest link sets the pace for the whole job (sizing math in Fat Trees).
• Reliability engineering. Meta's 16K-GPU Llama 3 run saw an unexpected interruption roughly every three hours (Llama 3 paper). Checkpoint and restart speed decide goodput (see Checkpoint Storage).
• Power swings. Synchronized compute/communicate phases make the whole facility's load oscillate. NVIDIA, Microsoft, and OpenAI have documented rack power swinging between roughly 30% and 100% in milliseconds (NVIDIA).
• Utilization is binary. A training cluster is either running the job or it isn't. Elasticity doesn't matter much.

Inference: it's a memory problem

Generation has two phases with very different profiles:

• Prefill processes the whole prompt in parallel. Big matrix multiplies, compute-bound, good GPU utilization.
• Decode generates one token per step per sequence. Every step has to read the active weights (and that sequence's KV cache) from HBM. Memory-bandwidth-bound.

Quick roofline check on an H100: ~989 TFLOPS dense BF16 against 3.35 TB/s HBM (NVIDIA H100) gives a ridge point around ~295 FLOPs per byte. Decode with BF16 weights does about 2 FLOPs per weight per sequence while reading 2 bytes per weight, so arithmetic intensity is roughly equal to the batch size. You need batches in the hundreds before decode becomes compute-bound. Every concurrent sequence also needs its own KV cache.

KV cache math

For Llama 3 70B: 80 layers, 8 KV heads (GQA), head dim 128, BF16.

KV bytes/token = 2 (K and V) × 80 × 8 × 128 × 2 bytes ≈ 320 KB

At 128K context, one sequence is ~40 GB, and 32 concurrent long-context users need ~1.3 TB of KV cache. That's more than the HBM in an 8-GPU H100 node, before you even load the weights. This is the core reason long-context serving is expensive and why HBM is the real constraint.

Why disaggregation took over

If prefill is FLOP-bound and decode is bandwidth-bound, running both on the same GPUs means one is always underusing the hardware. The fix is to split them into separate pools:

NVIDIA's Dynamo is the most visible implementation. Prefill and decode run on separate GPU pools, KV cache moves GPU-to-GPU over NVLink/RDMA through its NIXL transfer library, and KV blocks can be offloaded to host DRAM, SSD, or remote storage (Dynamo docs; KV cache offload). A KV-aware router sends requests to workers that already hold matching prefixes, so repeated system prompts and multi-turn chats skip prefill entirely.

Infra implications:

• Scale-up domain size matters for inference too. Large MoE models served with wide expert parallelism benefit from NVL72-class NVLink domains. All-to-all inside the rack is much cheaper than across the scale-out fabric.
• The fabric can be lighter. Inference traffic mostly stays inside a replica. KV transfers between prefill and decode pools need bandwidth, but you rarely need a full non-blocking 3-tier fabric across the whole fleet.
• Storage changes role. In inference, storage is part of the memory hierarchy (KV offload), not just a place to load weights from.

The economics are different too

	Training	Inference
Unit of value	Time-to-train a model	$/million tokens at a latency SLO
Utilization pattern	Near 100% while the job runs	Diurnal, bursty, needs headroom
Key hardware metric	FLOPS + scale-out bandwidth	HBM bandwidth + capacity
Biggest software lever	Parallelism strategy, fault tolerance	Batching, KV reuse, quantization, disaggregation
Contract shape	Large reserved blocks	Mix of reserved + elastic

Training clusters get bought in big reserved blocks and run flat out. Inference fleets need to follow demand. That's why many operators move hardware down the stack as it ages: last-gen training clusters become this year's inference capacity.

Bottom line

Don't build one cluster and pretend it's the other. Size training for synchronized throughput and recovery. Size inference for HBM bandwidth, KV capacity, and latency SLOs. And watch disaggregated serving closely, because it shipped and it's eating the old "one model, one node" approach.