Communication Dans Un Congrès Année : 2024

Cascade of phase transitions in the training of Energy-based models

Dimitrios Bachtis
  • Fonction : Auteur
Giulio Biroli
  • Fonction : Auteur
Beatriz Seoane

Résumé

In this paper, we investigate the feature encoding process in a prototypical energybased generative model, the Restricted Boltzmann Machine (RBM). We start with an analytical investigation using simplified architectures and data structures, and end with numerical analysis of real trainings on real datasets. Our study tracks the evolution of the model's weight matrix through its singular value decomposition, revealing a series of phase transitions associated to a progressive learning of the principal modes of the empirical probability distribution. The model first learns the center of mass of the modes and then progressively resolve all modes through a cascade of phase transitions. We first describe this process analytically in a controlled setup that allows us to study analytically the training dynamics. We then validate our theoretical results by training the Binary-Binary RBM on real datasets. By using datasets of increasing dimension, we show that learning indeed leads to sharp phase transitions in the high-dimensional limit. Moreover, we propose and test a mean-field finite-size scaling hypothesis. This shows that the first phase transition is in the same universality class of the one we studied analytically, and which is reminiscent of the mean-field paramagnetic-to-ferromagnetic phase transition.

Fichier principal
Vignette du fichier
RBMTransitionAndDynamics-4.pdf (3.05 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04897693 , version 1 (20-01-2025)

Identifiants

  • HAL Id : hal-04897693 , version 1

Citer

Dimitrios Bachtis, Giulio Biroli, Aurélien Decelle, Beatriz Seoane. Cascade of phase transitions in the training of Energy-based models. NeurIPS 2024 - 38th Annual Conference on Neural Information Processing Systems, Dec 2024, Vancouver, Canada. ⟨hal-04897693⟩
0 Consultations
0 Téléchargements

Partager

More