GPU clusters for AI: what your data center needs before installing the first H100
A GPU cluster for training or inference of large models looks nothing like a traditional general-compute cluster. Per-rack density jumps from 8 to 12 kW up to 60 to 100 kW, power draw is simultaneous rather than variable, and the internal network demands bandwidth most data centers have not sized for. Installing the first H100 without first resolving these three fronts is the most common cause of projects that slip six months and double their CAPEX.
This article describes what changes operationally when a data center pivots to AI workloads, which technical parameters are affected, and which design decisions are worth closing before energizing the first GPU rack. This is neither an NVIDIA manual nor a buyer’s guide: it is the punch list your infrastructure team must close before signing the contract with the accelerated-compute vendor.
Why an AI rack does not look like a traditional rack
A typical AI rack houses 4 to 8 compute nodes, each with 8 H100 GPUs or equivalent, plus host CPUs, system memory, NVMe (Non-Volatile Memory Express, high-speed SSD interface) storage, and a dedicated high-speed network. The immediate consequence is that the rack’s electrical density multiplies 5 to 10 times compared with a traditional general-compute rack.
The load profile also changes. In traditional compute, servers see daytime peaks of 20% to 30% above the mean. In training of large models, peaks are simultaneous and short-lived but they demand that the electrical system, the cooling, and the network all be sized to absorb them without throttling. What in traditional compute is an operating margin becomes, in AI, a hard constraint: a 100 ms voltage drop can abort a 72-hour training run.
What changes in power and cooling
The first visible change is the electrical service. An AI rack with 8 H100 nodes at full load can draw 80 to 100 kW, which requires dedicated three-phase circuits per rack, not the single-phase outlets that typically populate traditional data centers. The PDU (Power Distribution Unit) shifts from 16 to 32 A single-phase to 60 to 100 A three-phase per phase.
The second change is cooling. The 80 to 100 kW per rack density far exceeds what perimeter CRAC (Computer Room Air Conditioning, precision air conditioning units) units typically handle, generally 15 to 25 kW per rack in a conservative design. The natural transition is to liquid cooling — either direct-to-chip cold plate or chilled water distribution to the row — but there are intermediate options like rear-door heat exchangers or in-row cooling that can extend the life of a traditional data center before the full migration to liquid.
Network and topology: what AI demands
The internal network of an AI cluster is not a corporate network: it is a GPU-to-GPU interconnect that moves hundreds of GB/s between nodes during distributed training. It is built on technologies like InfiniBand NDR (Next Data Rate, 400 Gb/s per link) or 400G/800G Ethernet with RoCE (RDMA over Converged Ethernet, protocol that allows direct remote-memory access over Ethernet), and requires high-density, low-latency, non-blocking switches.
A typical design for 8 racks of H100s can include two switch tiers: a leaf tier with 8 ports of 400G per switch that connect directly to the compute nodes, and a spine tier that aggregates the leaves. The exact number of switches and ports depends on the parallel training algorithm being run, but the fiber and transceiver sizing to support 400G per link follows the rules discussed in the article on DAC, AOC, and single-mode fiber.
Five technical decisions you must take before installing the first H100
| Infrastructure area | Critical decision to document | Operational impact if postponed |
|---|---|---|
| Electrical capacity | Headroom in the electrical system to add 60 to 100 kW per rack without tripping service limits | Restriction to lower per-rack density or UPS expansion with 9 to 12 months of civil work |
| Cooling topology | Decide between cold plate, rear-door heat exchangers, or in-row cooling before the first node | Mixed topologies within the same cluster that complicate operations |
| Network and topology | Define InfiniBand vs 400G/800G Ethernet, number of switches per tier, and addressing plan | Costly re-cabling of entire zones of the data center |
| Physical layout and airflow | Dedicated zone with hot-aisle containment for the AI racks | Air recirculation and throttling within the first hour of operation |
| Monitoring and response plan | Per-node metrics (power, temperature, network telemetry, errors) with frequency and responsible owner | Outages without diagnosis and retraining costs from late failure discovery |
Common risks and how to avoid them
Four risks account for most projects that slip or blow budget when installing the first AI cluster:
- Undersizing the electrical service. This is the most frequent mistake. The decision is made with the initial configuration, but growth of the cluster 12-18 months in demands power that was not contemplated, and expanding a substation is civil work that can take 9-12 months in Mexico.
- Choosing the wrong network topology. InfiniBand NDR and 800G Ethernet have similar latencies for training workloads, but the tooling ecosystem, switch cost, and availability of trained staff differ significantly. The decision must align with the team’s talent strategy, not only with raw performance.
- Underestimating in-room heat load. An 80 kW rack generates the thermal equivalent of a small industrial oven. If the cooling system was not sized, the rack inlet-to-outlet temperature differential exceeds 15°C and triggers throttling within two hours of operation.
- Postponing the monitoring plan. Fine-grained telemetry is not optional: without per-node metrics, an individual GPU failure surfaces as a slow job rather than an alert. This multiplies the diagnostic time and triggers retraining costs when the failure is discovered late.
Sources
[1] NVIDIA — DGX Systems (data center AI compute platform) — https://www.nvidia.com/en-us/data-center/dgx-systems/
[2] NVIDIA — Data Center products overview — https://www.nvidia.com/en-us/data-center/
[3] IEA — Data Centres and Data Transmission Networks — https://www.iea.org/energy-system/digitalisation/data-centres-and-data-transmission-networks
[4] Uptime Institute — Research and reports — https://uptimeinstitute.com/resources/research-and-reports/
[5] Wikipedia — GPU cluster (background reference) — https://en.wikipedia.org/wiki/GPU_cluster
