AI workloads vs traditional workloads: the power, cooling, and network gap

For two decades, data center sizing followed stable rules: 5 to 10 kW per rack, PUE (Power Usage Effectiveness) between 1.4 and 1.8, switches with 10G or 40G uplinks, and a hierarchical network architecture that scaled by adding spines (network core) or leaves (access tiers). AI workloads — training of large models, large-scale inference, distributed fine-tuning — broke all four rules at once.

Rack density jumped from 10 kW to 60-100 kW at training sites, the load profile shifted from predictable to highly variable with simultaneous peaks, east-west traffic (between servers in the same cluster) replaced north-south as the dominant pattern, and the traditional hierarchical network architecture stopped sustaining the demand.

This article describes what changed in each of those four variables, why a data center designed for traditional loads cannot host AI loads without partial reconstruction, and what design decisions separate a site ready for AI from one trapped in a costly redesign.

Power: from kW to MW per rack

A traditional general-compute rack (web, databases, email) consumes between 5 and 10 kW with a predictable profile: daytime variations of 20-30% above the mean, planable with monthly history. A large-model training rack consumes between 60 and 100 kW — a 10x higher density — with a load profile where peaks coincide with the compute-intensive phases of training and drop during synchronization phases. According to the analyses published by the International Energy Agency (IEA) in its Electricity 2024 report, the aggregate electrical consumption of the data center sector is growing at rates above the average of total electricity demand, and the share attributable to AI loads is rising within that total.

The operational consequence is that a site ready for AI needs an electrical service entrance sized for megawatts per row, not for kilowatts per rack — and that changes the type of transformer, the cable gauge, the UPS capacity, and the backup architecture. NVIDIA documents in the technical sheets of the DGX family (accelerated computing systems) that a typical DGX H100 rack consumes on the order of 40-50 kW, and a multi-rack cluster scales linearly.

Cooling: the load profile that changed everything

The mechanical system of a traditional data center is sized for a continuous thermal load with margin. The mechanical system of an AI site has to dissipate heat concentrated in a small volume and absorb simultaneous thermal peaks when every node in the cluster starts a compute-intensive phase. This forces three mechanical design decisions to be reconsidered:

• Power density per rack. 60-100 kW per rack means the air extraction system must move air volumes 5-10x larger than in a traditional rack, or be supplemented with direct liquid cooling (DLC) at the chip level or rear-door heat exchangers.
• Operating temperature. Heat concentration reduces air’s capacity as a thermal transport medium. ASHRAE expanded the acceptable temperature and humidity range for modern IT equipment in its recent technical guidelines, but for AI racks that range is narrower: the equipment works, but with less thermal margin, which increases the risk of throttling (automatic speed reduction to avoid overheating) if the mechanical system fails.
• Mechanical redundancy. The profile of simultaneous thermal peaks reduces the average effect of N+1 redundancy (one backup component for each main component). At an AI site, the mechanical system typically requires 2N redundancy (complete duplicated systems) at the extraction stages, not just N+1.

Networking: where the traditional architecture breaks

The traditional hierarchical network architecture — 40G or 100G spines, leaves with 10G or 25G ports, uplinks to the spine via bundles of 4 or 8 links — was designed for north-south traffic where most of the data enters and leaves the data center.

Distributed training of large models inverts that proportion: every GPU in a cluster needs to communicate with every other GPU at each training step, generating east-west traffic (between servers in the same cluster) that can reach 10x or more of the north-south traffic.

The consequence is that hierarchical architectures saturated at their uplinks stop sustaining performance, and the industry is migrating to fabrics with high port-density 400G or 800G switches, three-tier Clos topologies (networks with multiple parallel paths) and direct GPU-to-GPU connectivity via NVLink (NVIDIA’s proprietary link) or InfiniBand (high-performance networking standard). A site designed for 10G/40G north-south cannot support 400G/800G east-west without rebuilding the switch plant and the trunk cabling.

Comparative table: traditional vs AI

| Variable | Traditional workload | AI workload |

The verdict: Adapt the infrastructure or fall into obsolescence

The gap between traditional loads and AI loads is not a gradual difference — it is a break along four simultaneous dimensions (power, thermal, network, redundancy) that forces the site to be rebuilt if it wants to host AI with growth headroom. The good news is that the industry regularly publishes roadmaps and reference architectures (NVIDIA DGX, hyperscaler architectures, IEA analyses) that serve as a starting point for making decisions. The bad news is that those decisions become more expensive with every year the site remains in its original architecture.

Sources

[1] IEA — Electricity 2024: https://www.iea.org/reports/electricity-2024

[2] Uptime Institute — Data Center Resources: https://uptimeinstitute.com/resources

[3] Uptime Institute — Blog: https://uptimeinstitute.com/blog

[4] ASHRAE Standard 90.1: https://www.ashrae.org/technical-resources/bookstore/standard-90-1

[5] Wikipedia — Hyperscale computing: https://en.wikipedia.org/wiki/Hyperscale_computing

[6] Wikipedia — Power usage effectiveness: https://en.wikipedia.org/wiki/Power_usage_effectiveness

[7] Wikipedia — Nvidia DGX: https://en.wikipedia.org/wiki/Nvidia_DGX

[8] NVIDIA — DGX Systems: https://www.nvidia.com/en-us/data-center/dgx-systems/

[9] NVIDIA — H100 Tensor Core GPU: https://www.nvidia.com/en-us/data-center/h100/

[10] Wikipedia — Data center: https://en.wikipedia.org/wiki/Data_center