Latency in data centers: why milliseconds matter more than you think

Latency is the distance a bit travels between leaving your server and returning with a response. In a modern corporate data center, you define that distance. And every extra meter has measurable consequences on revenue, experience, and cost.

Here’s what changes between milliseconds, and why the physical architecture of the DC is what drives the answer.

What latency is made of

The total latency of a link is not just the linear distance of the fiber. It is composed of physical and logical factors:

• Propagation speed in the medium: Light travels at nearly 300,000 km/s in vacuum. However, passing through the glass core of single-mode fiber, the material’s density reduces its speed to around 200,000 km/s. In copper, the speed drops even further, ranging between 150,000 and 180,000 km/s depending on the dielectric material of the cable.
• Latency per kilometer: This speed in glass translates to roughly 5 microseconds per kilometer of fiber (which equates to about 10 microseconds per kilometer on a round trip, or RTT). So a connection of just 10 kilometers already adds 100 microseconds purely from the light travel.
• Equipment latency: Each switch in the path adds between 1 and 10 microseconds, depending on its buffer size and whether it uses a cut-through (immediate forwarding) or store-and-forward (full packet processing) architecture. Modern NICs add 1 to 3 microseconds, while operating system stacks can add entire milliseconds.
• Queues and congestion: When a network port receives more traffic than its bandwidth can process, packets are forced to wait in the switch’s buffer memory. At this point, latency stops being a hardware physical limit and becomes an operational optimization problem.

Applications where the millisecond really matters

For most corporate workloads, milliseconds get diluted in the user experience. But there are exceptions where latency is the product:

1. High-frequency trading. A 1 ms improvement in the link to the exchange represents a measurable competitive advantage in revenue per operation.
2. Real-time AI inference. Language, vision, and speech recognition models have response times sensitive to latency between user and GPU. Every extra 100 ms degrades the perception of fluidity.
3. Telemedicine and remote surgery. Latency between the surgeon and the robot must stay below 20 ms for haptic control to be viable.
4. Competitive gaming and live streaming. Applications where competitive edge or quality of experience is measured in milliseconds.

How latency is controlled in the data center architecture

Three physical levers make the difference:

1. Fiber distance. The most basic decision: does the processing live 5 meters or 50 kilometers apart? That is decided at design time, not later.
2. Switching topology. A well-designed spine-leaf keeps latency between any pair of servers under 5 microseconds. Three-tier topologies (core/aggregation/access) multiply it.
3. Storage type. NVMe over PCIe transports in microseconds. SAN Fibre Channel adds stack latency. Rotational disk adds mechanical latency in milliseconds.

If your DC has time-sensitive workloads, the question is not “how much latency do I have?” but “where does every microsecond come from?” That’s where competitive edge is decided.