The algorithms used to train neural networks, like stochastic gradient descent (SGD), have
close parallels to natural processes that navigate a high-dimensional parameter space--for
example protein folding or evolution. Our study uses a Fokker-Planck approach, adapted
from statistical physics, to explore these parallels in a single, unified framework. We focus in
particular on the stationary state of the system in the long-time limit, which in conventional
SGD is out of equilibrium, exhibiting persistent currents in the space of network parameters …

The algorithms used to train neural networks, like stochastic gradient descent (SGD), have
close parallels to natural processes that navigate a high-dimensional parameter space--for
example protein folding or evolution. Our study uses a Fokker-Planck approach, adapted
from statistical physics, to explore these parallels in a single, unified framework. We focus in
particular on the stationary state of the system in the long-time limit. In contrast to its
biophysical analogues, conventional SGD leads to a nonequilibrium stationary state …