Collaborative Research: SCALE MoDL: Advancing Theoretical Minimax Deep Learning: Optimization, Resilience, and Interpretability

在过去的十年里，深度学习在广泛的社会和商业应用中取得了巨大的成功。然而，传统的深度学习依赖于用神经网络拟合数据，而神经网络会产生缺乏弹性的模型。例如，自动驾驶中使用的模型容易受到恶意攻击，例如，在停车标志上贴上艺术贴纸可能会导致模型将其归类为限速标志;已知面部识别中使用的模型偏向于特定种族或性别的人;医疗保健中的模型可以被黑客攻击以重建用于训练这些模型的患者的身份。下一代深度学习范式需要提供弹性模型，以提高对恶意攻击的鲁棒性，用户之间的公平性和隐私保护。该项目旨在开发一种全面的学习理论，以增强深度学习的模型弹性。该项目将产生快速算法和新的诊断工具，用于训练，增强，可视化和解释模型弹性，所有这些都具有广泛的研究和社会意义。研究活动还将对本科生和研究生产生积极的教育影响。该项目开发的材料将被整合到机器学习，统计和数据可视化课程中，并将使电气和计算机工程，统计，数学和计算机科学专业的跨学科学生受益。该项目将积极参与代表性不足的学生，并将研究与STEM本科生和研究生的教育相结合。 该项目还将为K-12学生制作用于工程夏令营的入门材料。在该项目中，研究人员将合作开发一个全面的极大极小学习理论，从优化，弹性和可解释性的角度推进对极大极小深度学习的基本理解。这些互补的理论发展，反过来，将指导设计新的极大极小学习算法，大大提高计算效率，统计保证和可解释性。这项研究包括三个主要方面。首先，研究人员将开发一种原则性的非凸极大极小优化理论，该理论支持可扩展、快速和收敛的梯度下降上升算法，用于训练复杂的极大极小深度学习模型。该理论将集中在分析的收敛速度和样本复杂度的算法。其次，研究人员将制定深度学习模型脆弱性的衡量标准，并研究minimaxity如何增强其对数据，模型和任务偏差的弹性。该理论将关注深度学习的统计极限。最后，研究人员将为一组新颖的可视化分析技术建立数学基础，以提高极大极小学习的模型可解释性。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The past decade has witnessed the great success of deep learning in broad societal and commercial applications. However, conventional deep learning relies on fitting data with neural networks, which is known to produce models that lack resilience. For instance, models used in autonomous driving are vulnerable to malicious attacks, e.g., putting an art sticker on a stop sign can cause the model to classify it as a speed limit sign; models used in facial recognition are known to be biased toward people of a certain race or gender; models in healthcare can be hacked to reconstruct the identities of patients that are used in training those models. The next-generation deep learning paradigm needs to deliver resilient models that promote robustness to malicious attacks, fairness among users, and privacy preservation. This project aims to develop a comprehensive learning theory to enhance the model resilience of deep learning. The project will produce fast algorithms and new diagnostic tools for training, enhancing, visualizing, and interpreting model resilience, all of which can have broad research and societal significance. The research activities will also generate positive educational impacts on undergraduate and graduate students. The materials developed by this project will be integrated into courses on machine learning, statistics, and data visualization and will benefit interdisciplinary students majoring in electrical and computer engineering, statistics, mathematics, and computer science. The project will actively involve underrepresented students and integrate research with education for undergraduate and graduate students in STEM. It will also produce introductory materials for K-12 students to be used in engineering summer camps.In this project, the investigators will collaboratively develop a comprehensive minimax learning theory that advances the fundamental understanding of minimax deep learning from the perspectives of optimization, resilience, and interpretability. These complementary theoretical developments, in turn, will guide the design of novel minimax learning algorithms with substantially improved computational efficiency, statistical guarantees, and interpretability. The research includes three major thrusts. First, the investigators will develop a principled non-convex minimax optimization theory that supports scalable, fast, and convergent gradient-descent-ascent algorithms for training complex minimax deep learning models. The theory will focus on analyzing the convergence rate and sample complexity of the developed algorithms. Second, the investigators will formulate a measure of vulnerability of deep learning models and study how minimaxity can enhance their resilience against data, model, and task deviations. This theory will focus on the statistical limits of deep learning. Lastly, the investigators will establish the mathematical foundations for a set of novel visual analytics techniques that increase the model interpretability of minimax learning. In particular, the theory will provide guidance on visualizing and interpreting model resilience.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.