---
title: "Training pays for backpropagation's backward pass; inference runs the forward pass alone — the backward pass costs roughly 2x the forward, making training tokens ~3x inference tokens"
type: "claim"
status: "budding"
audit_status: "capture-verified (Epoch AI pieces fetched directly at capture level, quotes verbatim per capture provenance; queen re-check not yet run) | 2026-09-11 audit (claude-fable-5-1, cross-model lane; writer unknown): both Epoch pieces re-fetched directly. source_quote and 'around three times more expensive' EXACT on source_url (Erdil, 2024-03-29; the backward-pass sentence is its footnote 1). The GPT-3 '3e23 FLOP … 3e11' and 'square root of the training compute' quotations are NOT on source_url — they are from Villalobos & Atkinson, 'Trading off compute in training and inference' (epoch.ai, 2023-07-28; footnote 10 and main text respectively), EXACT there; source_supporting_url added and the body now says which piece each quote is from. Discrepancy recorded inline: that 2023 piece's footnote 15 puts a backward pass at '3x as expensive as a forward pass', not Erdil's 2x — the headline's 2x rests on Erdil alone. Headline claim unchanged."
source_supporting_url: "https://epoch.ai/blog/trading-off-compute-in-training-and-inference"
source_supporting_note: "Villalobos & Atkinson (Epoch AI), 'Trading off compute in training and inference', 2023-07-28 — the home of the GPT-3 3e23/3e11 figures (footnote 10) and the square-root heuristic (main text); listed under source_author as 'supporting' since promotion, URL added 2026-09-11"
source_url: "https://epoch.ai/blog/optimally-allocating-compute-between-inference-and-training"
source_title: "Optimally allocating compute between inference and training"
source_author: "Ege Erdil (Epoch AI); supporting: Pablo Villalobos & David Atkinson (Epoch AI)"
source_date: "2024-03-29T00:00:00.000Z"
source_tier: 1
source_quote: "When training a model, we need to perform both forward and backward passes, and backward passes are generally twice as expensive to do as forward passes."
provenance: "Promotion from 10-inbox/raw/2026-07-06-is-training-vs-inference-compute-asymmetry-...md, 2026-07-07, queen cycle 4 — answering 50-questions/question-bridge-training-inference-compute.md (opened cycle 1, researched by the bee the same night)"
origin: "session"
date_created: "2026-07-07T00:00:00.000Z"
tags: ["training","inference","backward-pass","compute-economics","bridge-note","epoch-ai"]
drafted_in: ["2026-07-09-inference-inverted","inference-inverted"]
verified_verbatim: "2026-08-07 — source_quote matched verbatim (normalized) against a direct fetch of source_url by seek_verify (no model involved)"
seek_code_commit: "f424b5f"
audits: ["2026-09-11 claude-fable-5-1"]
---


The bridge between the vault's two largest clusters is a mechanism, stated
in FLOPs terms by Epoch AI: "When training a model, we need to perform both
forward and backward passes, and backward passes are generally twice as
expensive to do as forward passes" — hence "tokens seen during training are
around three times more expensive compared to tokens seen or generated
during inference" (Erdil 2024, the `source_url`). The scale of the
asymmetry, illustrated on GPT-3 in Epoch's earlier piece (Villalobos &
Atkinson 2023, `source_supporting_url`, footnote 10): "the cost of training
was 3e23 FLOP, whereas the cost of a single inference is 3e11" — one
training run ≈ a trillion inferences, with single-inference compute
"usually close to the square root of the training compute" (a heuristic
with stated variability, not a law). *(2026-09-11 audit: the two Epoch
pieces do not agree on the backward-pass multiplier — the 2023 piece's
footnote 15 says training "requires 3.5 trillion backward passes, each of
which is 3x as expensive as a forward pass", against Erdil's "twice as
expensive"; the headline's ~2x/~3x rests on Erdil. The order-of-magnitude
point survives either reading.)*

The backward pass is backpropagation's own computational content — the thing
[[claim-linnainmaa-reverse-mode-single-pass]] describes and
[[claim-reverse-mode-multiple-independent-discovery]] traces. Inference
([[claim-ai-inference-means-running-a-model]],
[[claim-llm-inference-prefill-decode]]) is the phase that never runs it.
That is the load-bearing link between the origins cluster and the economics
cluster.

**What this note deliberately does NOT claim:** that inference therefore
dominates industry compute. The magnitude story is separate and unsettled —
Epoch's theoretical allocation argument predicts *comparable* spending; the
one real datapoint (OpenAI 2024, Tier 2 via Epoch's curation of press-derived
figures) shows training spend *above* inference spend ($3B vs $1.8B). The
circulating "two-thirds" figure is ledgered at
[[myth-inference-two-thirds-of-compute]], and
[[claim-inference-dominant-ai-compute-2026]] keeps its [unverified-quant]
flag. See [[moc-inference-economics]], [[moc-backpropagation-origins]].