On December 14, 2020, Juncai He, a former postdoc at the Center for Computational Mathematics and Applications (CCMA), was invited to give a talk titled “Hierarchical and Multigrid Structures in Deep and Convolutional Neural Networks” on the Workshop on Mathematical Machine Learning and Application.
In this talk, He first showed how to utilize hierarchical basis methods to understand and interpret a special type of ReLU deep neural network (DNN) [1] for approximation quadratic and multiplication functions which plays a critically important role in a series of recent exponential approximation results of ReLU DNNs. During this procedure, he gave some unexpected representation properties of ReLU DNNs [2] and showed some exponential approximation results for both smooth and non-smooth functions [3]. Then, he presented a constrained linear model [4] that provides a different explanation for the feature extraction steps in ResNet type models. Furthermore, he demonstrated how a new type of convolutional neural network, known as MgNet [5], can be derived by making minor modifications to a classic geometric multigrid method for partial differential equations and then discussed the theoretical and practical potential of MgNet.
The Workshop on Mathematical Machine Learning and Application, which took place online in Zoom organized by the Center for the CCMA at the Pennsylvania State University during December 14-16, 2020, brought together more than 700 mathematicians from all over the world to share the greatest advances in this discipline that contributes so much to both education and science in general.
[1] Juncai He, Lin Li, Jinchao Xu and Chunyue Zheng. ReLU Deep Neural Networks and linear Finite Elements. Journal of Computational Mathematics, 38(3):502– 527, 2020.
[2] Juncai He, Lin Li and Jinchao Xu. ReLU deep neural networks from the perspective of hierarchical basis. In preparation, 2021.
[3] Juncai He, Lin Li and Jinchao Xu. Approximation properties of ReLU deep neural networks for smooth and non-smooth functions. In preparation, 2021.
[4] Juncai He, Yuyan Chen, Lian Zhang and Jinchao Xu. Constrained linear data-feature mapping for image classification. arXiv preprint arXiv:1911.10428.
[5] Juncai He and Jinchao Xu. MgNet: A unified framework of multigrid and convolutional neural network. Science China Mathematics, 62(7): 1331-1354, 2019.