Inproceedings,

Global optimality of Elman-type RNNs in the mean-field regime

, , and .
Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, page 196--227. PMLR, (23--29 Jul 2023)

Abstract

We analyze Elman-type recurrent neural networks (RNNs) and their training in the mean-field regime. Specifically, we show convergence of gradient descent training dynamics of the RNN to the corresponding mean-field formulation in the large width limit. We also show that the fixed points of the limiting infinite-width dynamics are globally optimal, under some assumptions on the initialization of the weights. Our results establish optimality for feature-learning with wide RNNs in the mean-field regime.

Tags

Users

  • @scadsfct

Comments and Reviews