Academy of Mathematics and Systems Science, CAS
Colloquia & Seminars

Speaker:	陈薇研究员，中国科学院计算技术研究所
Inviter:
Title:	Convergence and Implicit Regularization of Deep Learning Optimizers
Language:	Chinese
Time & Venue:	2023.04.11 10:00 N109
Abstract:	In this talk, I will present our recent work on the convergence/generalization analysis for the popular optimizers in deep learning. (1) We establish the convergence for Adam under (L0,L1 ) smoothness condition and argue that Adam can adapt to the local smoothness condition while SGD cannot. (2) We study the implicit regularization of DL optimizers. For adaptive optimizers, we prove that the convergent direction of RMSProp is the same with GD, while that of AdaGrad depends on the conditioner; for momentum acceleration, we prove that gradient descent with momentum converges to the L2 max-margin solution, which is the same as vanilla gradient descent.