CAREER: Understanding the Inductive Biases in Modern Machine Learning

现代机器学习（特别是深度学习）的最新进展正在引领人工智能时代，这有可能彻底改变我们日常生活的方方面面。然而，就像蒸汽机的早期一样，对深度学习的令人满意的理解到目前为止还难以捉摸。我们目前缺乏一个正式的深度学习理论，这个理论可以解释为什么我们可以用看似不够的训练数据训练过于复杂的模型，并且仍然找到推广到以前看不见的数据的解决方案，或者为什么为一个任务训练的模型在另一个相关任务上也表现良好，或者为什么训练过的模型如此容易受到轻微的、几乎难以察觉的数据损坏的影响。该项目旨在通过开发一种与实践紧密结合并受到实践推动的解释性和规定性深度学习理论来解决这一需求。该方法不是将学习简单地视为一个黑盒优化问题，而是通过揭示算法启发式来研究内部工作原理，算法启发式在赋予训练模型出色的泛化特性方面可能发挥同样重要的作用。鉴于深度学习的广泛适用性以及拟议研究中理论分析和实证研究的互补性，该项目特别适合将研究整合到教育和推广中。拟议的教育活动包括课程开发、暑期实习、黑客马拉松以及通过巴尔的摩当地项目开展的教师拓展活动。该项目研究了显式算法正则化在早期停止、批处理规范化和辍学形式中的作用，以及优化算法和网络架构的选择，以提供有助于泛化的适当归纳偏差。该项目的第二个首要目标是更广泛地理解深度学习中的泛化现象。它试图理解为什么记忆训练数据的系统仍然可以很好地泛化，神经网络架构如何实现迁移学习，以及如何设计健壮的算法，以保证深度学习解决方案在数据对抗性破坏的情况下泛化。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Recent advances in modern machine learning (deep learning in particular) are ushering in the era of artificial intelligence, which has the potential to revolutionize every aspect of our daily lives. However, much like the early days of the steam engine, a satisfactory understanding of deep learning has so far been elusive. We currently lack a formal theory of deep learning, one that could explain why we can train overly complex models with seemingly not enough training data and still find solutions that generalize to previously unseen data, or why models trained for one task also perform well on another related task, or why trained models are so vulnerable to slight, nearly imperceptible, corruptions of data. This project aims to address this need by developing an explanatory and prescriptive theory of deep learning that is tightly integrated with and motivated by the practice. Rather than view learning as simply a black-box optimization problem, the approach investigates the inner workings by shedding light on algorithmic heuristics that potentially play an equally important role in endowing the trained models with excellent generalization properties. Given the broad applicability of deep learning and the complementary nature of theoretical analyses and empirical studies in the proposed research, the project is particularly suited for integrating research into education and outreach. The proposed educational activities include curriculum development, summer internships, hackathons, and instructor's outreach through local Baltimore programs. The project investigates the role of explicit algorithmic regularization in the form of early stopping, batch normalization, and dropout, as well as the choice of optimization algorithms and network architecture in providing an adequate inductive bias that helps with generalization. A second overarching goal of the project is to understand, more broadly, the generalization phenomenon in deep learning. It seeks to understand why systems that memorize the training data can still generalize well, how the neural network architecture enables transfer learning, and how to design robust algorithms that will guarantee that deep learning solutions generalize despite adversarial corruption to data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.