Network fabric for GPU clusters: why InfiniBand keeps winning and when 800G Ethernet catches up

When you have a training cluster with 8, 16, or 64 GPUs, the network between GPUs becomes the immediate bottleneck. The choice between InfiniBand and 800G Ethernet affects latency, throughput, cost, and compatibility with frameworks like NCCL and RCCL. This is the technical decision that defines the real performance of your cluster, after choosing the GPUs themselves.

Why the cluster network is the critical factor

Distributed training of large models requires constant communication between GPUs. In every training step, GPUs exchange gradients — for a 70B-parameter model, that means tens of GB per step that have to cross the network in milliseconds. If the network takes longer, the entire step slows down, the GPUs sit idle waiting for data, and your hardware investment is wasted. A poorly dimensioned network can reduce the cluster’s effective throughput by 30–50%.

InfiniBand: the de facto standard

InfiniBand, in its current versions HDR (200 Gbps per port), NDR (400 Gbps per port), and XDR (800 Gbps per port), is the network used by most hyperscalers and research centers for AI training. Three reasons:

1. Sub-microsecond latency. InfiniBand has switch latency of 100–200 nanoseconds per hop. Comparable Ethernet has 500–800 nanoseconds. For frequent inter-GPU communications, that difference adds up.
2. Native Remote Direct Memory Access (RDMA). InfiniBand allows one GPU to read another GPU’s memory directly without going through an intermediate CPU. This eliminates a bottleneck in all-reduce operations like those used by NCCL.
3. Framework optimization. NVIDIA’s NCCL, AMD’s RCCL, and most distributed training frameworks are optimized for InfiniBand. Switching from InfiniBand to Ethernet typically requires re-tuning communication parameters.

800G Ethernet: the alternative that is catching up

Ethernet has advanced a lot in the last 3 years. The 400G and 800G versions, combined with RoCE (RDMA over Converged Ethernet), promise to close the gap with InfiniBand. Three considerations:

1. Ecosystem cost. Ethernet is commodity. Switches, optical transceivers (OSFP), and cabling have more vendors and more economies of scale. For large clusters, the cost per port can be 30–50% lower than InfiniBand.
2. Improved latency. With Priority Flow Control (PFC), Explicit Congestion Notification (ECN), and Data Center Bridging (DCB), modern Ethernet switches can deliver per-hop latencies of 400–600 nanoseconds — still more than InfiniBand but in the right order of magnitude for many workloads.
3. RoCE maturation. RoCE v2 over Ethernet enables RDMA with latencies close to InfiniBand. The main frameworks (NCCL, RCCL) support RoCE, although with more careful tuning than for native InfiniBand.

When InfiniBand is still the right answer

InfiniBand is the right decision in three scenarios:

1. Cluster with 64+ GPUs training a single model. All-reduce communication is constant and latency accumulates. InfiniBand NDR (400 Gbps) and XDR (800 Gbps) are the only options that deliver consistent latency at that scale; XDR also provides port-level bandwidth parity with 800G Ethernet, while preserving the sub-microsecond latency advantage of the InfiniBand ecosystem.
2. NVIDIA DGX or HGX cluster. These systems are certified and optimized for InfiniBand. Mixing with Ethernet requires manufacturer re-certification and can void support.
3. Multi-tenant workload. InfiniBand has better isolation between tenants via virtual lanes (VLs). In clusters where multiple teams train simultaneously, InfiniBand prevents one tenant from monopolizing the network.

When 800G Ethernet is already enough

800G Ethernet is sufficient in three scenarios:

1. Small cluster (≤32 GPUs) training medium-sized models. The performance gap between InfiniBand and Ethernet is small at this scale. The cost savings compensate.
2. Distributed inference with small batching. Inference parallelizes differently from training; traffic is less synchronous and tolerates more latency. 800G Ethernet performs well.
3. Integration with existing Ethernet infrastructure. If you already have a DC with 100/400G Ethernet switching, adding 800G Ethernet for the AI cluster simplifies operation and reduces the inventory of specialty switches.

Spine-leaf topology: what both options share

Beyond the InfiniBand vs Ethernet choice, there is one network topology that is standard for both: spine-leaf. In this architecture, the leaf switches connect the GPU racks and the spine switches form the backbone. Each leaf is connected to each spine, giving uniform latency between any pair of GPUs in the cluster. The choice of the number of spines (typically 2–8) depends on the aggregate bandwidth you need. For a 32-GPU cluster, 2 spines at 400G each are usually enough; for 128 GPUs, you need 4–8 spines with 800G uplinks.

How to avoid over-engineering

The most common mistake in new projects is over-dimensioning the network from day one. If your cluster starts with 16 GPUs and you plan to grow to 64, don’t buy 128-port switches. Start with modular switches that you can scale by adding leaves as you add racks. Initial capex drops 40–60% and the network scales with your actual demand. Operational complexity is slightly higher (more switches to manage) but the financial savings justify the trade-off in most cases.

Practical recommendation

If your cluster will have 64+ GPUs training large LLMs with NVIDIA, go with InfiniBand NDR (400 Gbps) or XDR (800 Gbps) — the tuning overhead for Ethernet is not justified at that scale, and XDR gives you exact port-level bandwidth parity versus 800G Ethernet with the latency advantage. If you have 16–32 GPUs and a limited budget, try 800G Ethernet with RoCE — current frameworks support it reasonably well. If your workload is inference or training of medium-sized models (≤13B), Ethernet is the pragmatic choice. The decision comes down to cluster size and workload model, not vendor preference.

Sources

1. NVIDIA — homepage — https://www.nvidia.com/en-us/data-center/
2. Ethernet Alliance — 800G specification overview — https://www.ethernetalliance.org/technology/800g/
3. Uptime Institute — Tier Classification framework (data center network standards) — https://uptimeinstitute.com/tier-certification
4. Arista Networks — homepage — https://www.arista.com/