Colossus
xAI, United States (2024) — 100,000 x NVIDIA H100 SXM (80GB)
8 × H100 SXM (80GB) per server
8 servers per rack
= 64 accelerators per rack, drawing 44.8 kW (accelerators only)
Rack layout Reported
Colossus uses Supermicro liquid-cooled 4U servers with 8 GPUs each, 8 servers per rack (64 GPUs/rack), grouped into 512-GPU arrays — per Supermicro's and xAI's public descriptions of the build.
1,563 racks holding 12,500 servers and 100,000 accelerators.
At roughly 3 m² per rack including aisle space, that's about 4,689 m² of floor — roughly 11 basketball courts. Floor area is estimated: real halls also carry cooling, power, and networking equipment this doesn't count.
If fully dedicated, this cluster could train...
| Known training run | Time on this cluster | Originally took |
|---|---|---|
| GPT-3 (175B) | 2.2 hours | duration not published |
| Llama 2 (70B) | 5.9 hours | 35.0 days on 2,048 x A100 SXM (80GB) |
| Llama 3.1 (405B) | 11.1 days | 54.0 days on 16,384 x H100 SXM (80GB) |
| PaLM (540B) | 18.0 hours | duration not published |
| Chinchilla (70B) | 4.0 hours | duration not published |
Hypothetical: assumes the whole cluster runs one job at the vendor-default MFU, with no downtime, restarts, or contention — Estimated. "Originally took" is the run's own reported hardware and duration where published, for contrast.
Source: xAI's own announcement (Sept 2024): Colossus (Memphis, TN) came online with 100,000 NVIDIA H100 GPUs, described at the time as the largest AI training supercomputer in single-cluster operation. xAI has since announced plans to expand Colossus toward 200,000+ GPUs (mixing in H200s); this entry reflects the initial, most solidly reported 100K figure rather than later expansion targets.