---
title: "Mandt, Hoffman and Blei (2017) model constant-learning-rate SGD near a loss minimum as an Ornstein–Uhlenbeck process"
type: "claim"
status: "seedling"
source_url: "https://arxiv.org/abs/1704.04289"
source_title: "Stochastic Gradient Descent as Approximate Bayesian Inference"
source_author: "Stephan Mandt, Matthew D. Hoffman, David M. Blei"
source_date: 2017
source_venue: "Journal of Machine Learning Research 18 (2017); 'Stochastic Gradient Descent as Approximate Bayesian Inference'"
source_quote: "Stochastic Gradient Descent with a constant learning rate (constant SGD) simulates a Markov chain with a stationary distribution."
source_tier: 1
audit_status: "capture-verified (arXiv 1704.04289 referenced at capture time; queen independent re-check blocked in headless promotion. The capture itself flagged this as the load-bearing 'SGD-as-OU' mechanism and advised double-checking before publication; the OU claim is an approximation under stated assumptions — see [[question-verify-mandt-2017-constant-sgd-ornstein-uhlenbeck]]). Kept seedling. | AUDIT 2026-09-12 (claude-fable-5-1, cross-model): arXiv:1704.04289 re-read via extract_pdf (v2, JMLR 18:1–35, 35 pp., TLS verified) — source_quote EXACT (abstract, opening sentence). The routed question sits in 50-questions/_answered/ but was closed on 2026-08-07 by seek_verify's mechanical quote-match only; its substantive items are discharged by this read: §3.2 'SGD as an Ornstein-Uhlenbeck Process' approximates constant SGD explicitly by the multivariate OU process (Eq. 11, citing Uhlenbeck & Ornstein 1930) under four stated assumptions — (1) Gaussian gradient noise via the central limit theorem, (2) noise covariance approximately constant near the optimum, (3) the finite-difference update approximated by the SDE, (4) loss locally quadratic with positive-definite Hessian — and the paper itself scopes the picture to the phase 'near the local optimum' where 'constant SGD begins to sample from its stationary distribution', stating that in the early search phase 'assumptions 1–4 are often violated'. The body's assumption list (constant learning rate, locally quadratic loss, roughly Gaussian noise) matches the paper. No correction; status: seedling unchanged — raising it is a writer's call now that mechanism and scope are read at primary."
provenance: "Promotion from 10-inbox/raw/2026-07-11-hop-stasis-is-an-ou-process.md, 2026-07-12"
origin: "batch"
derived_from: "10-inbox/raw/2026-07-11-hop-stasis-is-an-ou-process.md"
writer_model: "claude-opus-4-8"
date_created: "2026-07-12T00:00:00.000Z"
tags: ["machine-learning","stochastic-gradient-descent","ornstein-uhlenbeck","optimization","loss-landscape","mean-reversion","approximate-bayesian-inference"]
verified_verbatim: "2026-08-07 — source_quote matched verbatim (normalized) against a direct fetch of source_url by seek_verify (no model involved)"
seek_code_commit: "89bc9f4"
audits: ["2026-09-12 claude-fable-5-1"]
---


In "Stochastic Gradient Descent as Approximate Bayesian Inference" (JMLR 2017), Mandt, Hoffman and Blei observe that "Stochastic Gradient Descent with a constant learning rate (constant SGD) simulates a Markov chain with a stationary distribution." Rather than converging to a point, constant-rate SGD keeps bouncing around a loss minimum, and the distribution of where it lands settles into a fixed stationary form. Their analysis approximates this Markov chain by a continuous-time Ornstein–Uhlenbeck (OU) process: near a minimum the loss is locally quadratic, so the gradient supplies a linear restoring force toward the bottom of the well while the minibatch noise plays the role of the driving randomness. SGD near a minimum is, on this account, OU mean-reversion in a quadratic basin.

This closes a bridge onto the vault's ML-optimization cluster. SGD's own lineage runs through stochastic approximation ([[claim-robbins-monro-1951-stochastic-approximation]]) and the LMS/stochastic-descent siblings ([[claim-amari-stochastic-descent-and-widrow-hoff-lms-are-sgd-siblings]]); its failure modes and geometry appear in [[claim-vanishing-gradient-chain-rule-pathology]] and [[claim-amari-1998-natural-gradient-fisher-steepest-descent]], catalogued under [[moc-backpropagation-origins]] and [[backpropagation-gap]]. The same OU restoring-force object governs fossil stasis ([[claim-ou-model-recasts-stasis-as-active-mean-reversion-to-an-optimum]]) and interest rates ([[claim-ornstein-uhlenbeck-process-links-brownian-motion-and-the-vasicek-model]]); the three legs are joined in [[observation-mean-reversion-to-an-optimum-recurs-across-fossil-stasis-bonds-and-sgd]].

The OU description is an *approximation* resting on stated assumptions — a constant learning rate, a locally quadratic loss, and roughly Gaussian gradient noise. The capture flagged this as the load-bearing mechanism of its whole cross-domain bridge and advised double-checking before publication; the note therefore stays `seedling`, with the scope and assumptions of the approximation routed to [[question-verify-mandt-2017-constant-sgd-ornstein-uhlenbeck]].