Academy of Mathematics and Systems Science, CAS
Colloquia & Seminars

Speaker:	张海樟教授, 中山大学数学学院(珠海)
Inviter:	许志强研究员
Title:	Convergence Thery of Deep Neural Networks: Arbitrary Activation Functions and Pooling
Language:	Chinese
Time & Venue:	2022.12.08 16：00-17：00 腾讯会议: 478-1365-3406
Abstract:	https://meeting.tencent.com/dm/3WoH5WwglQOj Deep neural networks, as a powerful system to represent high dimensional complex functions, play a key role in deep learning. Convergence of deep neural networks is a fundamental issue in building the mathematical foundation for deep learning. We investigated the convergence of deep ReLU networks and deep convolutional neural networks in two recent papers, where only the Rectified Linear Unit (ReLU) activation function was studied. The important pooling strategy was not considered therein either. In this paper, we study the convergence of deep neural networks as the depth tends to infinity for general activation functions which cover most of commonly-used activation functions in artificial neural networks. Pooling will also be studied. Specifically, we adapt the linear method developed in our recent papers to prove that the major condition there is still sufficient for the neural networks defined by non-expansive activation functions to converge, despite their nonlinearity. For contractive activation functions such as the logistic sigmoid function, we establish a uniform and exponential convergence of the associated deep neural networks.