CDS&E: Applied Geometry and Harmonic Analysis in Deep Learning Regularization: Theory and Applications

CDS

• 批准号: 2052525
• 负责人: Wei Zhu
• 金额: $ 5.16万
• 依托单位: University of Massachusetts Amherst
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-06-30 至 2024-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2052525&HistoricalAwards=false
• 关键词: CDS&E; Applied; Geometry; Harmonic; Analysis

In this era of Big Data, deep learning has become a burgeoning domain with immense potential to advance science, technology, and human life. Despite the tremendous practical success of deep neural networks (DNNs) in various data-intensive machine learning applications, there still remain many open problems to be addressed: (1) DNNs tend to suffer from overfitting when the available training data are scarce, which renders them less effective in the small data regime. (2) DNNs have shown to have the capability of perfectly “memorizing” random training samples, making them less trustworthy when the training data are noisy and corrupted. (3) While symmetry is ubiquitous in machine learning (e.g., in image classification, the class label of an image remains the same if the image is spatially rescaled and translated,) generic DNN architectures typically destroy such symmetry in the representation, which leads to significant redundancy in the model to “memorize” such information from the data. The goal of this project is to address these challenges in deep learning by exploiting the low-dimensional geometry and symmetry within the data and their network representations, aiming at developing new theories and methodologies for deep learning regularization that can lead to tangible advances in machine learning and artificial intelligence especially in the small/corrupted data regime. In addition the project also provides research training opportunities for graduate students.The overarching theme of this project is to leverage recent progress in mathematical methods from differential geometry and applied harmonic analysis to improve the stability, reliability, data-efficiency, and interpretability of deep learning. This will involve the development of both foundational theories and efficient algorithms to achieve the following three objectives: (1) The development of manifold-based DNN regularizations with significantly improved generalization performance by focusing on the topology and geometry of both the input data and their representations. This will unlock the potential of deep learning in the small data regime. (2) Establishing and analyzing an innovative framework of imposing geometric constraint in deep learning that has immense potential of limiting the memorizing capacity of DNN. The mathematical analysis of the training dynamics of such model will shed light on the understanding of the fundamental difference between “memorization” and generalization in deep learning. (3) The construction of deformation robust symmetry-preserving DNN architectures for various symmetry transformations on different data domains. By "hardwiring" the symmetry information into the deformation robust representations, the regularized DNN models will have improved performance and interpretability with reduced redundancy and model size. In terms of application, the project will demonstrate and deploy the proposed theories in real-world machine learning tasks, such as object recognition, localization, and segmentation. The techniques developed in this project will be widely applicable across different disciplines, providing fundamental building blocks for the next generation of mathematical tools for the computational modeling of Big Data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在这个大数据时代，深度学习已经成为一个新兴领域，具有推动科学、技术和人类生活的巨大潜力。尽管深度神经网络（dnn）在各种数据密集型机器学习应用中取得了巨大的实际成功，但仍然存在许多有待解决的开放性问题：(1)当可用的训练数据稀缺时，dnn往往会遭受过拟合，这使得它们在小数据体系中效果较差。(2) dnn具有完美“记忆”随机训练样本的能力，这使得它们在训练数据有噪声和损坏时不那么可信。(3)虽然对称性在机器学习中无处不在（例如，在图像分类中，如果图像在空间上被重新缩放和翻译，图像的类标签保持不变），但通用DNN架构通常会破坏表示中的这种对称性，这导致模型中存在显着的冗余，以“记忆”数据中的此类信息。该项目的目标是通过利用数据及其网络表示中的低维几何和对称性来解决深度学习中的这些挑战，旨在为深度学习正则化开发新的理论和方法，从而在机器学习和人工智能方面取得切实进展，特别是在小/损坏的数据体系中。此外，该项目还为研究生提供研究培训机会。该项目的总体主题是利用微分几何和应用谐波分析的数学方法的最新进展来提高深度学习的稳定性、可靠性、数据效率和可解释性。这将涉及基础理论和有效算法的发展，以实现以下三个目标：(1)通过关注输入数据及其表示的拓扑和几何结构，开发基于流形的DNN正则化，显著提高泛化性能。这将释放深度学习在小数据领域的潜力。(2)建立并分析了在深度学习中施加几何约束的创新框架，该框架具有限制深度神经网络记忆能力的巨大潜力。对该模型训练动态的数学分析将有助于理解深度学习中“记忆”和“泛化”之间的根本区别。(3)针对不同数据域上的各种对称变换，构造形变鲁棒的保对称DNN结构。通过将对称信息“硬连接”到变形鲁棒表示中，正则化DNN模型将在减少冗余和模型大小的情况下提高性能和可解释性。在应用方面，该项目将在现实世界的机器学习任务中演示和部署所提出的理论，如物体识别、定位和分割。该项目开发的技术将广泛应用于不同学科，为下一代大数据计算建模的数学工具提供基础构建模块。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Deformation Robust Roto-Scale-Translation Equivariant CNNs

• 发表时间: 2021-11
• 期刊: ArXiv
• 作者: Liyao (Mars) Gao;Guang Lin;Wei Zhu

A Dictionary Approach to Domain-Invariant Learning in Deep Networks

• 发表时间: 2019-09
• 期刊: arXiv: Learning
• 作者: Ze Wang;Xiuyuan Cheng;G. Sapiro;Qiang Qiu

Adversarial defense via the data-dependent activation, total variation minimization, and adversarial training

• DOI: 10.3934/ipi.2020046
• 发表时间: 2021
• 期刊: Inverse Problems & Imaging
• 影响因子: 1.3
• 作者: Bao Wang;A. Lin;Penghang Yin;Wei Zhu;A. Bertozzi;S. Osher

Structure-preserving GANs

• 发表时间: 2022-02
• 作者: Jeremiah Birrell;M. Katsoulakis;Luc Rey-Bellet;Wei Zhu

Graph convolution with low-rank learnable local filters”. International Conference on Learning Representations

具有低阶可学习局部滤波器的图卷积。

• 发表时间: 2021
• 期刊: International Conference on Learning Representations
• 作者: Cheng, Xiuyuan;Miao, Zichen;Qiu, Qiang
• 通讯作者: Qiu, Qiang