--- title: "The KV cache grows with context length and can balloon to several times the size of the model itself" type: "claim" status: "seedling" audit_status: "flagged (V-006) | 2026-09-11 audit (claude-fable-5-1, cross-check): both Redis quotations re-verified exact-match at the cited URL (Wallace, 2026-04-28, updated 2026-04-29). The V-006 mechanism gap is now discharged at Tier 1: Pope et al., 'Efficiently Scaling Transformer Inference' (arXiv:2211.05102, Google, 2022-11-09; extract_pdf sha256 a34aacd15d1d4918da0001697a01b4ae8f547fec0169fe886f5f10a529529a93, TLS verified), §2.1: 'for batch size 512 and context length 2048, the KV cache totals 3TB, which is 3 times the size of the model's parameters. The on-chip memory needs to load this KV cache from off-chip memory once for every token generated during which the computational core of the chip is essentially idle' — both the growth-past-model-size claim and the memory-bandwidth-bound decode mechanism are stated there. One new defect: the body's Pragmatic Engineer quotation ('the cached results of the attention algorithm, reused between requests to speed up inference') has no URL, and the obvious candidate (Orosz & Morikawa, 'Scaling ChatGPT: Five Real-World Engineering Challenges', 2024-02-20) does not contain the phrase on fetch — marked [unverified-quote — needs direct read] per the quote-provenance rule. Original flag retained below; status stays seedling." flags: ["[unverified-mechanism — needs primary] KV-cache growth/memory-bandwidth mechanism rests on a Tier-3 vendor blog (quotes verified verbatim against the source, but sources.md requires Tier 1-2 for technical-mechanism claims). Retro-fitted 2026-07-06 per Cali ruling 7 (audit V-006).","[RESOLVED 2026-09-11] The mechanism flag above is discharged: Tier-1 corroboration at Pope et al. 2022 (arXiv:2211.05102) §2.1 — KV cache 3× model parameters at batch 512 / context 2048 on PaLM 540B, loaded from HBM every generated token. See corroborating_sources and the body section added 2026-09-11.","[unverified-quote — needs direct read] The Pragmatic Engineer sentence in 'What it enables' has no URL and was not found at the candidate article on 2026-09-11. Added 2026-09-11 audit."] corroborating_sources: [{"url":"https://arxiv.org/abs/2211.05102","title":"Efficiently Scaling Transformer Inference","author":"Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, Jeff Dean (Google)","date":"2022-11-09","tier":1,"sha256":"a34aacd15d1d4918da0001697a01b4ae8f547fec0169fe886f5f10a529529a93","quote":"For a 500B+ model with multihead attention, the attention KV cache grows large: for batch size 512 and context length 2048, the KV cache totals 3TB, which is 3 times the size of the model's parameters. The on-chip memory needs to load this KV cache from off-chip memory once for every token generated during which the computational core of the chip is essentially idle.","added":"2026-09-11 audit"}] date_created: "2026-06-04T00:00:00.000Z" provenance: "Seek research batch, 2026-06-04" tags: ["LLM","KV-cache","inference","memory","GPU","VRAM","attention"] source_url: "https://redis.io/blog/prefill-vs-decode/" source_title: "Prefill vs Decode: LLM Inference Phases Explained" source_author: "Jim Allen Wallace" source_date: "2026-04-28" source_tier: 3 related_notes: ["claim-llm-inference-prefill-decode","claim-ai-inference-means-running-a-model","claim-inference-engineering-emerged-as-specialty"] drafted_in: ["1968-had-no-word","2026-07-13-1968-had-no-word"] audits: ["2026-09-11 claude-fable-5-1"] --- The **key-value cache** (KV cache) is the central memory structure of [[LLM inference]]. It stores the intermediate results of the [[attention mechanism]] — specifically, the key and value vectors computed for every token that the model has seen — so those computations need not be repeated on each decode step. "Every decode step depends on every prior token, so the model has to remember the full context. That memory is the KV cache. It starts at the size of your prompt and grows by one entry per generated token." (Jim Allen Wallace, Redis, 2026-04-28) ## Why it matters for memory pressure At production scale, the KV cache is not a marginal structure. "At scale, with long responses across many concurrent requests, the cache can balloon to several times the size of the model itself. Every decode step has to read all of that, which is a big reason decode is memory-bandwidth-bound." (Jim Allen Wallace, Redis, 2026-04-28) This means that as [[long-context window|context windows]] expand — from 4K tokens in 2022 to 200K and beyond by 2025 — the KV cache grows proportionally, compounding the [[memory bandwidth]] bottleneck in the [[decode phase]]. ## What it enables Without the KV cache, each decode step would require recomputing attention over the *entire* sequence from scratch, making inference complexity quadratic in context length. The KV cache reduces this to approximately linear per token, by caching the attention state built during [[prefill]]. The Pragmatic Engineer describes the KV cache as "the cached results of the attention algorithm, reused between requests to speed up inference." [unverified-quote — needs direct read: no URL recorded, and the phrase was not found at the candidate article (Orosz & Morikawa, 2024-02-20) on 2026-09-11] ## Primary corroboration (added 2026-09-11 audit) The vendor-blog claim has a Tier-1 antecedent. Pope et al. (Google, arXiv:2211.05102, 2022), analysing PaLM inference on TPU v4, state in §2.1: "For a 500B+ model with multihead attention, the attention KV cache grows large: for batch size 512 and context length 2048, the KV cache totals 3TB, which is 3 times the size of the model's parameters. The on-chip memory needs to load this KV cache from off-chip memory once for every token generated during which the computational core of the chip is essentially idle." The same section gives the crossover: "At small batch sizes and sequence lengths, the time to load weights dominates. At larger batch sizes and sequence lengths (e.g. 2048+ tokens with batch size 512+), the time to load the KV cache dominates." That is the growth-past-model-size claim and the memory-bandwidth mechanism in the primary literature, four years before the Redis post. ## Engineering implications Managing the KV cache is one of the central problems in inference infrastructure: - **Batching strategy**: Multiple concurrent requests share GPU memory, competing for KV cache space. - **Cache eviction**: Long-running contexts may need to be paged to slower storage (CPU RAM or disk) and retrieved. - **Prefix caching**: If many requests share the same system prompt prefix, the KV vectors for that prefix can be cached and reused across requests, reducing both latency and compute. The KV cache is thus both the mechanism that makes [[autoregressive generation]] tractable and the primary reason that long-context, high-concurrency inference is memory-dominated rather than compute-dominated. See [[claim-llm-inference-prefill-decode]] for the two-phase framing. See also: [[claim-llm-inference-prefill-decode]], [[claim-inference-engineering-emerged-as-specialty]], [[claim-ai-inference-means-running-a-model]]