CDS&E: Applied Geometry and Harmonic Analysis in Deep Learning Regularization: Theory and Applications

点击翻译按钮获取中文摘要

In this era of Big Data, deep learning has become a burgeoning domain with immense potential to advance science, technology, and human life. Despite the tremendous practical success of deep neural networks (DNNs) in various data-intensive machine learning applications, there still remain many open problems to be addressed: (1) DNNs tend to suffer from overfitting when the available training data are scarce, which renders them less effective in the small data regime. (2) DNNs have shown to have the capability of perfectly “memorizing” random training samples, making them less trustworthy when the training data are noisy and corrupted. (3) While symmetry is ubiquitous in machine learning (e.g., in image classification, the class label of an image remains the same if the image is spatially rescaled and translated,) generic DNN architectures typically destroy such symmetry in the representation, which leads to significant redundancy in the model to “memorize” such information from the data. The goal of this project is to address these challenges in deep learning by exploiting the low-dimensional geometry and symmetry within the data and their network representations, aiming at developing new theories and methodologies for deep learning regularization that can lead to tangible advances in machine learning and artificial intelligence especially in the small/corrupted data regime. In addition the project also provides research training opportunities for graduate students.The overarching theme of this project is to leverage recent progress in mathematical methods from differential geometry and applied harmonic analysis to improve the stability, reliability, data-efficiency, and interpretability of deep learning. This will involve the development of both foundational theories and efficient algorithms to achieve the following three objectives: (1) The development of manifold-based DNN regularizations with significantly improved generalization performance by focusing on the topology and geometry of both the input data and their representations. This will unlock the potential of deep learning in the small data regime. (2) Establishing and analyzing an innovative framework of imposing geometric constraint in deep learning that has immense potential of limiting the memorizing capacity of DNN. The mathematical analysis of the training dynamics of such model will shed light on the understanding of the fundamental difference between “memorization” and generalization in deep learning. (3) The construction of deformation robust symmetry-preserving DNN architectures for various symmetry transformations on different data domains. By "hardwiring" the symmetry information into the deformation robust representations, the regularized DNN models will have improved performance and interpretability with reduced redundancy and model size. In terms of application, the project will demonstrate and deploy the proposed theories in real-world machine learning tasks, such as object recognition, localization, and segmentation. The techniques developed in this project will be widely applicable across different disciplines, providing fundamental building blocks for the next generation of mathematical tools for the computational modeling of Big Data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.