Collaborative Research: SCALE MoDL: Advancing Theoretical Minimax Deep Learning: Optimization, Resilience, and Interpretability

过去的十年见证了深度学习在广泛的社会和商业应用中取得的巨大成功。然而，传统的深度学习依赖于用神经网络来拟合数据，众所周知，这会产生缺乏弹性的模型。例如，自动驾驶中使用的模型容易受到恶意攻击，例如，在停车标志上贴上艺术贴纸会导致模型将其归类为限速标志；面部识别中使用的模型已知偏向特定种族或性别的人；医疗保健中的模型可以被黑客攻击，以重建用于训练这些模型的患者的身份。下一代深度学习范式需要提供具有弹性的模型，以提高对恶意攻击的健壮性、用户之间的公平性和隐私保护。本项目旨在开发一种全面的学习理论，以增强深度学习的模型弹性。该项目将产生快速算法和新的诊断工具，用于培训、增强、可视化和解释模型复原力，所有这些都可以具有广泛的研究和社会意义。研究活动还将对本科生和研究生产生积极的教育影响。该项目开发的材料将被整合到机器学习、统计和数据可视化的课程中，并将造福于电气和计算机工程、统计、数学和计算机科学专业的跨学科学生。该项目将积极吸收代表性不足的学生参与，并将研究与STEM本科生和研究生的教育相结合。在这个项目中，研究人员将合作开发一个全面的极小极大学习理论，从最优化、弹性和可解释性的角度推进对极小极大深度学习的基本理解。这些互补的理论发展反过来将指导新的极小极大学习算法的设计，大大提高了计算效率、统计保证和可解释性。这项研究包括三个主要推动力。首先，研究人员将开发一种原则性的非凸极小极大优化理论，该理论支持用于训练复杂的极小极大深度学习模型的可伸缩、快速和收敛的梯度下降-上升算法。该理论将重点分析所开发算法的收敛速度和样本复杂度。其次，研究人员将制定深度学习模型脆弱性的衡量标准，并研究最小化如何增强其对数据、模型和任务偏差的弹性。这一理论将侧重于深度学习的统计极限。最后，研究人员将为一套新的视觉分析技术奠定数学基础，这些技术可以增加极小极大学习的模型可解释性。特别是，该理论将在可视化和解释模型弹性方面提供指导。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The past decade has witnessed the great success of deep learning in broad societal and commercial applications. However, conventional deep learning relies on fitting data with neural networks, which is known to produce models that lack resilience. For instance, models used in autonomous driving are vulnerable to malicious attacks, e.g., putting an art sticker on a stop sign can cause the model to classify it as a speed limit sign; models used in facial recognition are known to be biased toward people of a certain race or gender; models in healthcare can be hacked to reconstruct the identities of patients that are used in training those models. The next-generation deep learning paradigm needs to deliver resilient models that promote robustness to malicious attacks, fairness among users, and privacy preservation. This project aims to develop a comprehensive learning theory to enhance the model resilience of deep learning. The project will produce fast algorithms and new diagnostic tools for training, enhancing, visualizing, and interpreting model resilience, all of which can have broad research and societal significance. The research activities will also generate positive educational impacts on undergraduate and graduate students. The materials developed by this project will be integrated into courses on machine learning, statistics, and data visualization and will benefit interdisciplinary students majoring in electrical and computer engineering, statistics, mathematics, and computer science. The project will actively involve underrepresented students and integrate research with education for undergraduate and graduate students in STEM. It will also produce introductory materials for K-12 students to be used in engineering summer camps.In this project, the investigators will collaboratively develop a comprehensive minimax learning theory that advances the fundamental understanding of minimax deep learning from the perspectives of optimization, resilience, and interpretability. These complementary theoretical developments, in turn, will guide the design of novel minimax learning algorithms with substantially improved computational efficiency, statistical guarantees, and interpretability. The research includes three major thrusts. First, the investigators will develop a principled non-convex minimax optimization theory that supports scalable, fast, and convergent gradient-descent-ascent algorithms for training complex minimax deep learning models. The theory will focus on analyzing the convergence rate and sample complexity of the developed algorithms. Second, the investigators will formulate a measure of vulnerability of deep learning models and study how minimaxity can enhance their resilience against data, model, and task deviations. This theory will focus on the statistical limits of deep learning. Lastly, the investigators will establish the mathematical foundations for a set of novel visual analytics techniques that increase the model interpretability of minimax learning. In particular, the theory will provide guidance on visualizing and interpreting model resilience.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.