Generalization at the Edge of Stability

Mario Tuci1,2,*, Caner Korkmaz2,*, Umut Şimşekli1,†, Tolga Birdal2,†
1 École Normale Supérieure, INRIA, CNRS   2Imperial College London
*,†Equal contribution
Code arXiv
Teaser image
Generalization at the Edge of Stability (EoS). Modeling stochastic optimization as a random dynamical system (RDS), we show that at EoS the leading sharpness satisfies \(\lambda_1>0\) implying expansion along at least one direction. The fundamental balance between expansion and contraction implies that the effective dimensionality of the dynamics, measured by our Sharpness Dimension (\(\mathrm{SD}\)), is strictly smaller than the ambient parameter space: \(\mathrm{SD}<d\). We prove that the worst-case generalization error is governed by \(\mathbf{SD}\) rather than the parameter count. Our results identify EoS as precisely the regime where generalization is controlled by a provably lower-dimensional attractor, providing a principled explanation for why overparameterized models can generalize beyond classical complexity measures.

Abstract

Training modern neural networks often relies on large learning rates, operating at the edge of stability, where the optimization dynamics exhibit oscillatory and chaotic behavior. Empirically, this regime often yields improved generalization performance, yet the underlying mechanism remains poorly understood. In this work, we represent stochastic optimizers as random dynamical systems, which often converge to a fractal attractor set (rather than a point) with a smaller intrinsic dimension. Building on this connection and inspired by Lyapunov dimension theory, we introduce a novel notion of dimension, coined the 'sharpness dimension', and prove a generalization bound based on this dimension. Our results show that generalization in the chaotic regime depends on the complete Hessian spectrum and the structure of its partial determinants, highlighting a complexity that cannot be captured by the trace or spectral norm considered in prior work. Experiments across various MLPs and transformers validate our theory while also providing new insights into the recently observed phenomenon of grokking.

BibTeX


        @misc{tuci2026generalizationedgestability,
          title={Generalization at the Edge of Stability}, 
          author={Mario Tuci and Caner Korkmaz and Umut Şimşekli and Tolga Birdal},
          year={2026},
          eprint={2604.19740},
          archivePrefix={arXiv},
          primaryClass={cs.LG},
          url={https://arxiv.org/abs/2604.19740}, 
    }