AI infrastructure for data centers: what changes in power, cooling and network when you train LLMs

When a data center moves from running inference or general workloads to training large AI models, the design assumptions you had stop working. Rack density multiplies, payload times change, and consumption peaks become recurring events. This is what changes operationally — and what you have to redesign if your client asks you to host LLM training.

Power: from 7 kW to 70 kW per rack

A traditional 1U server consumes 200-400W. A GPU training server (NVIDIA H100 in DGX configuration) consumes 5-10 kW per node, and a rack full of them reaches 70-100 kW. That density requires three changes: a dedicated high-amperage circuit per rack, a UPS sized for the new peak load, and a generator capable of responding to the training consumption ramp. The breaker for a dense GPU rack must be 3-Phase 100A or more — very different from the typical 30A of conventional racks.

Cooling: why air stops working

At 70 kW per rack, the conventional air system simply does not move enough heat. The thermal extraction capacity of a typical CRAC unit is 20-30 kW per unit. For GPU racks, you need two routes: (a) direct-to-chip liquid cooling (DLC) for the hotspots, where the CDU carries the heat outside; or (b) rear-door heat exchangers that complement air cooling. Option (a) has higher initial capex but reduces cooling consumption by 30-40%. Option (b) is cheaper but requires more aisle space and does not handle peaks as efficiently.

Network: from 10 GbE to 200+ GbE

Distributed LLM training requires moving terabytes of gradients between GPUs at every training step. A conventional 10 GbE network is the immediate bottleneck. The real options are: 100 GbE (InfiniBand HDR) or 200/400 GbE (NDR, Quantum-2). This is not a switch upgrade — it is a completely different network architecture with single-mode fiber optic cabling, high-density switches, and a leaf-spine topology that minimizes latency. Migrating from 10 GbE to 200 GbE is typically a 6-12 month project and $500k-2M MXN depending on size.

The power spikes nobody anticipates

Training workloads have a unique consumption pattern: high consumption during the compute step (forward + backward pass) and low consumption during data loading and checkpointing. That means the data center sees oscillations of 30-50% of total consumption in cycles of seconds. The UPS must respond to those oscillations without transferring to battery (which would degrade the batteries quickly). The generator must be sized to sustain the peak, not just the average. And the contract with CFE must account for the real maximum demand, not the average — because CFE charges for peak demand, not total consumption.

The physical layout changes

GPU racks do not mix well with storage racks or general server racks. The heat and noise from GPU fans make a mixed aisle problematic for maintenance. The trend is to physically separate: an ‘AI cluster’ in one zone of the DC with its own CDU/cooling, its own high-speed network, and its own UPS circuit. This allows operating the AI zone with different practices (higher density, different maintenance cadence) without affecting the rest of the DC.

The energy factor: where consumption concentrates

In a typical training cluster, the energy distribution looks like this: 65-75% in the GPUs, 8-12% in host CPUs, 5-8% in memory and SSD, 5-10% in networking (switches, optical transceivers), and the rest in cooling, PSU and losses. When you optimize for AI training, the GPU is your biggest lever — but also your biggest heat source. That is why cooling becomes the operational bottleneck: if you do not extract heat from the GPUs fast enough, throttling mechanisms lower the frequency and your step time degrades.

The difference between training and inference

It is worth restating the distinction because commercial pitches confuse it. Training is the process of adjusting model weights using large datasets — it consumes a lot of compute and energy, typically in cycles of hours to weeks. Inference is using an already-trained model to respond to prompts — it consumes far less, typically milliseconds to seconds per response. A DC can host inference without redesigning (1-3 kW per server, compatible with most architectures). Training requires the complete redesign described in this article. If your client says ‘we want to do AI’ and has a 500 kW DC, ask whether they want to train or only run inference. The technical and investment implications are completely different.

When NOT to host LLM training

If your current DC does not have at least 2 MW of available electrical capacity and 1 MW of cooling, the cost of enabling the site for training exceeds that of migrating the workload to a hyperscale operator. Inference of trained models (inference) can be hosted in mid-sized DCs — it consumes 1-3 kW per server and is compatible with most current architectures. But training, no. That distinction is the one most often confused in commercial pitches.

Practical recommendation

Before signing a hosting contract for LLM training, ask your provider for three documents: (a) a specific layout of the AI zone with its CDU and its dedicated network; (b) the electrical and cooling capacity available for that zone, with declared peaks; (c) the capacity and uptime history of the segment, not of the entire DC. If the provider cannot deliver those three documents, the project will not execute well.

Sources

1. NVIDIA — HGX data center platform (GPU density reference) — https://www.nvidia.com/en-us/data-center/hgx/
2. Vertiv — homepage — https://www.vertiv.com/
3. Uptime Institute — AI infrastructure considerations (industry report) — https://uptimeinstitute.com/ai-services/ai-infrastructure-advisory
4. NVIDIA — DGX platform (training infrastructure) — https://www.nvidia.com/en-us/data-center/dgx-platform/