TPU v4 Pod (single pod)
Google, United States (2021) — 4,096 x Google TPU v4
4 × TPU v4 per server
16 servers per rack
= 64 accelerators per rack, drawing 12.3 kW (accelerators only)
Rack layout Reported
Google, 'TPU v4: An Optically Reconfigurable Supercomputer' (ISCA 2023): 64 TPU v4 chips per rack, arranged as a 4x4x4 cube.
64 racks holding 1,024 servers and 4,096 accelerators.
At roughly 3 m² per rack including aisle space, that's about 192 m² of floor — about 46% of a basketball court. Floor area is estimated: real halls also carry cooling, power, and networking equipment this doesn't count.
If fully dedicated, this cluster could train...
| Known training run | Time on this cluster | Originally took |
|---|---|---|
| GPT-3 (175B) | 7.2 days | duration not published |
| Llama 2 (70B) | 19.2 days | 35.0 days on 2,048 x A100 SXM (80GB) |
| Llama 3.1 (405B) | 867.7 days | 54.0 days on 16,384 x H100 SXM (80GB) |
| PaLM (540B) | 58.5 days | duration not published |
| Chinchilla (70B) | 13.2 days | duration not published |
Hypothetical: assumes the whole cluster runs one job at the vendor-default MFU, with no downtime, restarts, or contention — Estimated. "Originally took" is the run's own reported hardware and duration where published, for contrast.
Source: Google, 'TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning' (ISCA 2023): each TPU v4 Pod contains up to 4,096 chips. This is the documented capacity of a single standard pod, not Google's total fleet — Google operates many pods (PaLM alone used two, see the validated-runs table above).