张海樟教授, 中山大学数学学院(珠海)

https://meeting.tencent.com/dm/3WoH5WwglQOj

Deep neural networks, as a powerful system to represent high dimensional complex functions, play a key role in deep learning. Convergence of deep neural networks is a fundamental issue in building the mathematical foundation for deep learning. We investigated the convergence of deep ReLU networks and deep convolutional neural networks in two recent papers, where only the Rectified Linear Unit (ReLU) activation function was studied. The important pooling strategy was not considered therein either. In this paper, we study the convergence of deep neural networks as the depth tends to infinity for general activation functions which cover most of commonly-used activation functions in artificial neural networks. Pooling will also be studied. Specifically, we adapt the linear method developed in our recent papers to prove that the major condition there is still sufficient for the neural networks defined by non-expansive activation functions to converge, despite their nonlinearity. For contractive activation functions such as the logistic sigmoid function, we establish a uniform and exponential convergence of the associated deep neural networks.