Spend the budget: parameters vs. tokens
From Chapter 45 of Large Language Models from the Ground Up. A training budget is a fixed number of FLOPs, and C ≈ 6·N·D ties model size N to token count D. One slider reallocates the same budget along that constraint. The loss traces a U-shaped valley whose floor — the compute-optimal point — sits near Chinchilla's famous 20 tokens per parameter.
Reallocate a fixed FLOPs budget under C = 6·N·D
Pick a budget, then drag the slider from "few big-model tokens" to "many small-model tokens." N and D trade off so C stays fixed (D = C / 6N). The estimated loss follows the Chinchilla parametric form; the valley floor lands in the mid-to-high teens, which is what Chinchilla's ≈20 rule of thumb rounds off. Famous models are marked by where they sit on the tokens-per-parameter axis.
The valley floor is compute-optimal: for this budget, no other split of size-vs-data gives lower loss, and it lands at D/N ≈ 20 — Chinchilla's rule, which said a 70B model wants 1.4 trillion tokens and beat the 4×-larger Gopher on the same compute. The left wall is the 2020 mistake: huge, undertrained giants like GPT-3 (175B on 300B tokens, under 2 tokens/param) and Gopher (≈1:1). The right wall is too small a model to use the data — yet notice where Llama-3-8B sits (≈1,900 tokens/param): far past optimal on purpose, because a model is served at ≈2·N FLOPs per token forever, so a smaller model that costs a little more loss saves inference compute on every request for its whole life. "Compute-optimal" only minimizes training loss; deployment changes the answer. (Loss uses the Chinchilla parametric form L = E + A/Nα + B/Dβ. The coefficients are the corrected ones from Besiroglu et al. 2024, whose replication found Chinchilla's originally published fit inconsistent with the paper's own headline — those numbers put the optimum at 50–122 tokens/param. The corrected fit lands at 15–19, which is why the green line drifts a little as you change budget instead of sitting on a round 20.)