- Meta says its largest AI clusters now outgrow a single region, so cable distance and network buffers have become design problems.
- Renting cloud GPUs adds engineering work, because providers differ in network design, and Meta requires providers to run a common test suite before it takes over a cluster.
- If you rent AI capacity, ask for acceptance test results, the network type in use, and which parts of the stack are open.
The network is becoming part of the computer
When thousands of chips must work as one machine, the cables and switches between them stop being plumbing. This is WebPulse's view of what Meta's latest engineering post shows. The network becomes part of the machine, and its limits become the machine's limits.
Meta's Network Infrastructure team published the post on October 12 for the Open Compute Project (OCP) Global Summit. It is Meta's own account, and it contains no independent measurements. It still gives a rare, detailed look at what a very large AI buyer worries about.
Faster switches in a cooled rack
Meta introduced IcePack, its first liquid-cooled network rack. Inside sit IceCube switch blades. Each is a 102.4 terabit-per-second switch with 128 ports of 800 gigabits. Meta says this doubles the bandwidth of its 51.2T platforms.
IceCube comes in two versions, one built on a Broadcom chip and one on an NVIDIA chip. Both run FBOSS, Meta's open-source network operating system. Meta's stated reason for the liquid cooling is that switches must fit within increasingly dense power and cooling limits.
Distance becomes a design constraint
Meta says its Prometheus site reaches roughly 1 gigawatt in 2026. Its Hyperion site is scaling toward 5 gigawatts. At that size, Meta says, a single cluster is bigger than any one region can hold.
Meta's answer is a layer called BAG. It joins many smaller fabrics into one cluster across buildings, campuses and regions. Between two regions, Meta reports, the links can move tens of petabits of data every second.
Distance then causes a physical problem. Meta explains that lossless Ethernet, which must not drop data, needs buffer space that grows with cable length. Think of stopping distance for a car: a longer road needs more room to stop. Once links run tens of kilometers, Meta calls buffer management a first-order problem.
Instead of planning every link for the worst case, FBOSS now measures each cable in place. It times probe traffic to work out the round-trip distance and sets the buffer from that. Meta also says much of the software that coordinates GPUs does not account for distance, so performance can fall off a cliff. Meta is working on better routing and on scheduling that keeps traffic from crossing regions.
Renting capacity has a hidden engineering bill
The most useful passage for buyers is about leased and cloud capacity. Meta says data centers take years to build while demand moves in months, so it uses leased facilities and public cloud alongside its own campuses.
Leasing helps only if a GPU performs alike in every building, whoever owns it. Meta says cloud providers differ in several ways: fabric layouts, buffer depth and the transport used for fast GPU-to-GPU memory access. Some providers use InfiniBand, a different technology from the Ethernet Meta favors. A setup that works on a few rented instances can also fail once it grows to a whole fleet, Meta warns.
Meta's fix is a common test suite. Before Meta takes over a cluster, the provider must run a shared set of benchmarks, starting at tens of GPUs and growing past a thousand. The aim is measured results, not promises. Meta's principle: heterogeneity is manageable, but standardization makes it affordable. Every difference between providers becomes engineering work.
That figure comes from Meta's new multiplane design. Each GPU has eight separate links, and each link runs to a different switch on its own network plane. The older design used a single-plane uplink instead. Meta developed the new one with NVIDIA and is working with joint cloud partners to standardize it.
Open standards as a cost argument
Meta also described Tahan, an Ethernet switch that links accelerators inside a rack. Such links have traditionally used proprietary technology, according to Meta. It says Tahan is shipping today. It also cites ESUN, an OCP effort on Ethernet for this job, with AMD, Arista, Broadcom, Cisco, Microsoft, NVIDIA and others.
A new OCP project, the Open Switch Platform Architecture (OSPA), is being led jointly by Meta, Microsoft and Alibaba. Its target is the switch platform layer. Meta says each new hardware platform needs months of duplicated custom work to bring up. It also open-sourced its MetaRoCE transport.
Meta has its own interests here, since open specs widen the pool of suppliers it can buy from. Still, the cost logic holds for any buyer: proprietary layers mean more work every time hardware changes.
Questions to put to your team
Most readers will not build a gigawatt cluster. Many will rent AI capacity. WebPulse draws four questions from Meta's own practice.
First, does the provider run standard acceptance tests before handover, and will it share the results? Second, which network and transport does the capacity use, and does it differ by region? Third, if a job spans sites, how does it behave when one part fails? Fourth, which parts of the stack are open specifications, and which tie you to one vendor?
The lesson is plain. The chips get the headlines, but the space between them is where costs and surprises collect. Ask for proof before you pay for performance.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Meta Engineering.
