The estimator
Compute, from first principles

* Estimator duration uses a default MFU per chip vendor, since most of these sources don't publish an MFU directly — hence its Estimated label even where the underlying compute figure is Reported. PaLM is the exception: Google directly reported their achieved MFU, so its estimator duration is Calculated from that reported figure instead of an assumed one.

GPT-3 (175B): reportedFlops is directly published by OpenAI. The paper does not state GPU count or wall-clock duration, so chipCount (1024x V100) here is an illustrative assumption based on commonly-cited estimates for clusters of that era, not an OpenAI-published figure — trainingDays/energy/cost derived from it should be read as illustrative only. (Brown et al. 2020, 'Language Models are Few-Shot Learners' (GPT-3 paper), Appendix D / Table D.1: total training compute of ~3.14x10^23 FLOPs (3640 petaflop/s-days).)

Llama 2 (70B): reportedChipHours (GPU-hours) is directly published by Meta. reportedFlops is not stated as a single number in the paper, so it's left undefined here and only the estimator's 6ND calculation is shown for it. The 2048-GPU cluster size is an assumed typical scale (per Meta's published RSC infrastructure), used only to convert the reported GPU-hours into an illustrative wall-clock duration (~35 days); the chip-hours total itself does not depend on this split. (Touvron et al. 2023, 'Llama 2: Open Foundation and Fine-Tuned Chat Models', Table 2: 1,720,320 A100-80GB GPU-hours for the 70B model.)

Llama 3.1 (405B): Parameter count, token count, chip count, training duration, and total FLOPs are all directly reported by Meta — the most fully-specified of the three runs. (Meta AI, 'The Llama 3 Herd of Models' (2024): 405B model pre-trained on up to 16K H100-80GB GPUs, ~3.8x10^25 FLOPs, over ~54 days.)

PaLM (540B): The only run here where MFU itself is directly reported rather than assumed — Google explicitly measured and published their achieved hardware utilization, which is the norm this project otherwise has to estimate for every other run (including its own live estimator). (Chowdhery et al. 2022, 'PaLM: Scaling Language Modeling with Pathways': 540B model, 780B tokens, trained on 6,144 TPU v4 chips (two Pods of 3,072) with a reported training efficiency (model FLOPs utilization) of 46.2%.)

Chinchilla (70B): reportedFlops is directly stated in the paper (it's the foundational result behind the 6ND-style compute-optimal scaling laws this whole estimator relies on), but the paper doesn't specify a chip count/training duration for this particular model, so chipCount and chip choice (TPU v4, DeepMind's typical training hardware in this era) here are illustrative assumptions only. (Hoffmann et al. 2022, 'Training Compute-Optimal Large Language Models' (the Chinchilla paper): 70B model, 1.4T tokens, ~5.76x10^23 FLOPs of training compute.)