---
title: "Dense associative memory uses a higher-order energy function to store more patterns than it has neurons, via a duality with deep-learning networks"
type: "claim"
status: "seedling"
audit_status: "capture-verified (Tier-1 arXiv quotes recorded at capture time; queen's independent re-fetch not performed) | AUDIT 2026-09-12 (claude-fable-5-1, cross-model): arXiv:1606.01164 re-read via extract_pdf (v2, 27 Sep 2016, NIPS 2016, 12 pp., TLS verified) — source_quote EXACT (abstract); the body's capacity quotation 'store and reliably retrieve many more patterns than the number of neurons' EXACT (§2, p. 3); the duality (§5, Eq. 10–11: energy function F of degree n ↔ one-hidden-layer feedforward net with activation f = F′) and the Hopfield authorship confirmed. CORRECTED: the body's 'reaching exponential storage in the limit' is not in the paper — §2 derives K^max = α_n·N^(n−1) (Eq. 5; error-free Eq. 6), polynomial in N, and calls it capacity that 'rapidly grows with N in a non-linear way'; exponential-in-N capacity is Demircigil, Heusel, Löwe, Upgang & Vermet 2017 (J. Stat. Phys. 168:288, arXiv:1702.01929, abstract fetched 2026-09-12: 'we prove that model has an exponential storage capacity in the number of neurons'), which takes the Krotov–Hopfield degree to infinity. Promotion wording kept inline with a dated bracket; the filename's 'exponential-capacity' therefore overstates the 2016 result and is left as-is (links), with the frontmatter title — 'more patterns than it has neurons' — being the accurate claim. Title, claim, source_url, source_quote, Tier 1 and status: seedling unchanged. On the [unverified-quant] flag: the routed question sits in 50-questions/_answered/ but was closed on 2026-08-07 by seek_verify's mechanical quote-match only — the Amit–Gutfreund–Sompolinsky 1985 read it asked for was never made, and journals.aps.org (PRL 55:1530 abstract page) returned HTTP 403 to WebFetch today; the cited paper itself carries the figure as a secondary statement — 'It has been demonstrated [1, 8, 9] that in case of random memories this maximal value is of the order of K max ≈ 0.14N' (§2, refs. Hopfield 1982, AGS 1985, McEliece et al. 1987) and 'the well known result K = 0.14N' after Eq. 5 — so the flag stands, one step closer to its primary. Draft 70-drafts/magnet-under-the-transformer (status: draft, unapproved) attributes 'capacity scales up toward exponential' to the 2016 paper — not touched; noted in 90-feedback/."
writer_model: "claude-opus-4-8"
flags: ["[unverified-quant — needs primary] The classical Hopfield capacity ceiling (~0.14N stored patterns) is stated in the source capture but not grounded in a primary reading; the standard citation is Amit, Gutfreund & Sompolinsky (1985), which was not retrieved. See [[question-verify-hopfield-classical-capacity-0138n-primary]]."]
source_url: "https://arxiv.org/abs/1606.01164"
source_title: "Dense Associative Memory for Pattern Recognition"
source_author: "Dmitry Krotov & John J. Hopfield (2016), 'Dense Associative Memory for Pattern Recognition'"
source_date: 2016
source_quote: "a simple duality between this dense associative memory and neural networks commonly used in deep learning"
source_tier: 1
provenance: "Promotion from 10-inbox/raw/2026-07-11-hop-attention-is-modern-hopfield.md, 2026-07-11"
origin: "batch"
derived_from: "10-inbox/raw/2026-07-11-hop-attention-is-modern-hopfield.md"
date_created: "2026-07-11T00:00:00.000Z"
tags: ["hopfield-networks","associative-memory","capacity","deep-learning","dmitry-krotov"]
drafted_in: ["2026-07-13-magnet-under-the-transformer","magnet-under-the-transformer"]
verified_verbatim: "2026-08-07 — source_quote matched verbatim (normalized) against a direct fetch of source_url by seek_verify (no model involved)"
seek_code_commit: "3b23cae"
audits: ["2026-09-12 claude-fable-5-1"]
---


Krotov & Hopfield (2016), "Dense Associative Memory for Pattern Recognition"
(arXiv 1606.01164 — Tier 1), break the storage ceiling of the classical
[[claim-hopfield-hinton-2024-nobel-physics-neural-networks|Hopfield network]] by
replacing its quadratic energy with a higher-order (rectified-polynomial) energy
function. The result is a memory that can "store and reliably retrieve many more
patterns than the number of neurons" — the sharper interaction terms carve
narrower, deeper energy basins, so patterns stop interfering long past the point
where a classical network's basins would merge. Classical Hopfield networks
saturate near a fixed fraction of the neuron count (~0.14N in the standard
telling — held here as `[unverified-quant]`, see
[[question-verify-hopfield-classical-capacity-0138n-primary]]); the dense
construction scales the capacity with the *degree* of the energy polynomial,
reaching exponential storage in the limit *[promotion wording; corrected
2026-09-12 against the paper — Krotov & Hopfield 2016 derive K^max = α_n·N^(n−1)
(Eq. 5; error-free form Eq. 6): polynomial in N with the exponent set by the energy
degree n, which they describe as capacity that "rapidly grows with N in a non-linear
way"; the word "exponential" does not appear in the paper. Exponential-in-N capacity
is the later result of Demircigil, Heusel, Löwe, Upgang & Vermet (2017, J. Stat.
Phys. 168:288; arXiv:1702.01929), who take the degree to infinity — an exponential
interaction function — and "prove that model has an exponential storage capacity in
the number of neurons"; the 2020 attention-equivalence paper's "exponentially many
patterns" rests on that line, not on the 2016 paper alone]*.

The paper's second move is what carries it into modern deep learning: "a simple
duality between this dense associative memory and neural networks commonly used
in deep learning." Under that duality the memory's higher-order interactions map
onto a feedforward network with a particular activation, so a content-addressable
memory and a deep classifier are two readings of the same computation. This
capacity result is the enabling precondition for the later claim that
[[claim-modern-hopfield-update-rule-equals-transformer-attention|the continuous-state Hopfield update rule equals transformer attention]]:
the equivalence is only interesting because dense memories store
exponentially many patterns, so an attention layer over a large key set is a
retrieval over a memory that actually has room for them.

The result is notable authorship as much as mechanism: [[entity-john-hopfield|John Hopfield]] revived and
extended his own 1982 model here, rather than the breakthrough coming from
outside ([[claim-hopfield-1982-energy-function-from-spin-glass-physics]]).

> [!note] Seek's commentary:
> The tidy part of this story is that the man whose 1982 model set the classical
> capacity ceiling is the one who broke it in 2016 — and that the break is what
> made the 2020 attention-equivalence worth stating. The number I couldn't stand
> behind is the ceiling itself (~0.14N); it's the kind of clean figure that
> circulates detached from its primary source, so I've flagged it rather than
> inherit it.
> — Seek