Deep Neural Networks for Structured Data: Regression, Distribution Estimation, and Optimal Transport

在过去的十年中，深度学习在现实世界的应用中取得了惊人的突破，包括计算机视觉、自然语言处理、语音识别、医疗保健和机器人技术。深度学习使用多层线性变换，然后是非线性激活来表示数据中的抽象。 人们普遍认为，深度神经网络擅长学习隐藏在数据集中的各种几何结构，如丰富的局部对称性、全局对称性或重复模式。然而，几乎没有理论可以解释深度神经网络在分析包含几何结构的复杂数据集方面的强大功能。该项目将开发理论和计算基础，以更好地了解深度神经网络如何利用数据集中的几何结构并实现出色的性能。研究结果将为开发新的深度学习模型和方法提供新的见解。该项目侧重于三组相关但不同的问题。第一组集中在使用深度神经网络对低维流形上的函数进行有效逼近。现有的理论表明，深度学习估计器在高维中收敛到真实函数的速度非常慢。当函数在低维流形上得到支持时，PI计划根据流形的内在维数证明快速收敛速度。该项目将在函数逼近理论、统计回归和分类中的误差分析以及深度学习的自适应理论方面做出贡献。第二组问题涉及由深度生成模型支持的低维流形上的概率分布的估计。这些模型利用两个神经网络来最小化估计量和数据分布之间的积分概率度量（IPM），在由深度生成器网络生成的分布类上。IPM中的功能类是通过一个深层的嵌入式网络来实现的。该项目将设计适当的生成器和嵌入式系统的网络架构，并证明深度生成模型的性能保证。第三组问题将集中在使用深度神经网络有效计算两个概率分布之间的最佳传输。在将最优运输问题重新表述为由两个神经网络参数化的最小-最大优化问题后，PI提出了一种原始对偶随机梯度下降算法来解决它。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

In the past decade, deep learning has made astonishing breakthroughs in real-world applications, including for example, computer vision, natural language processing, speech recognition, healthcare, and robotics. Deep learning uses multiple layers of linear transformations followed by nonlinear activations to represent abstractions in the data. It is a common belief that deep neural networks are good at learning various geometric structures hidden in data sets, such as rich local regularities, global symmetries, or repetitive patterns. However, little theory has been established to explain the power of deep neural networks for analyzing complex data sets containing geometric structures. This project will develop the theoretical and computational foundations to better understand how deep neural networks exploit geometric structures in data sets and achieve outstanding performance. The results will provide new insights on developing new deep learning models and methodologies.This project focuses on three sets of related but distinct problems. The first set focuses on efficient approximation of functions on low-dimensional manifolds using deep neural networks. Existing theories show that deep learning estimators converge to the true function extremely slowly in high dimensions. When the function is supported on a low dimensional manifold, the PIs plan to prove a fast convergence rate depending on the intrinsic dimension of the manifold. This project will make contributions in function approximation theory, error analysis in statistical regression and classification, and adaptive theory of deep learning. The second set of problems concerns estimation of probability distributions supported on a low-dimensional manifold by deep generative models. These models utilize two neural networks to minimize the Integral Probability Metric (IPM) between the estimator and the data distribution, over the class of distributions generated by a deep generator network. The function class in IPM is realized by a deep discriminator network. This project will design proper network architectures of the generator and the discriminator, and prove performance guarantees of deep generative models. The third set of problems will focus on efficiently computing the optimal transport between two probability distributions using deep neural networks. After reformulating the optimal transport problem as a min-max optimization problem parametrized by two neural networks, the PIs propose a primal dual stochastic gradient descent algorithm to solve it.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.