CAREER: Understanding the Inductive Biases in Modern Machine Learning

职业：理解现代机器学习中的归纳偏差

• 批准号: 1943251
• 负责人: Raman Arora
• 金额: $ 50万
• 依托单位: Johns Hopkins University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-02-15 至 2025-01-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1943251&HistoricalAwards=false
• 关键词: CAREER; Understanding; Inductive; Biases; Modern

Recent advances in modern machine learning (deep learning in particular) are ushering in the era of artificial intelligence, which has the potential to revolutionize every aspect of our daily lives. However, much like the early days of the steam engine, a satisfactory understanding of deep learning has so far been elusive. We currently lack a formal theory of deep learning, one that could explain why we can train overly complex models with seemingly not enough training data and still find solutions that generalize to previously unseen data, or why models trained for one task also perform well on another related task, or why trained models are so vulnerable to slight, nearly imperceptible, corruptions of data. This project aims to address this need by developing an explanatory and prescriptive theory of deep learning that is tightly integrated with and motivated by the practice. Rather than view learning as simply a black-box optimization problem, the approach investigates the inner workings by shedding light on algorithmic heuristics that potentially play an equally important role in endowing the trained models with excellent generalization properties. Given the broad applicability of deep learning and the complementary nature of theoretical analyses and empirical studies in the proposed research, the project is particularly suited for integrating research into education and outreach. The proposed educational activities include curriculum development, summer internships, hackathons, and instructor's outreach through local Baltimore programs. The project investigates the role of explicit algorithmic regularization in the form of early stopping, batch normalization, and dropout, as well as the choice of optimization algorithms and network architecture in providing an adequate inductive bias that helps with generalization. A second overarching goal of the project is to understand, more broadly, the generalization phenomenon in deep learning. It seeks to understand why systems that memorize the training data can still generalize well, how the neural network architecture enables transfer learning, and how to design robust algorithms that will guarantee that deep learning solutions generalize despite adversarial corruption to data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

现代机器学习（尤其是深度学习）的最新进展正在迎来人工智能时代，它有可能彻底改变我们日常生活的各个方面。然而，就像早期的蒸汽机一样，迄今为止对深度学习的令人满意的理解还难以实现。我们目前缺乏一种正式的深度学习理论，它可以解释为什么我们可以用看似不够的训练数据来训练过于复杂的模型，并且仍然找到泛化到以前未见过的数据的解决方案，或者为什么为一项任务训练的模型在另一项相关任务上也表现良好，或者为什么训练后的模型如此容易受到轻微的、几乎难以察觉的数据损坏的影响。该项目旨在通过开发一种与实践紧密结合并受实践驱动的深度学习的解释性和规范性理论来满足这一需求。该方法并没有将学习视为简单的黑盒优化问题，而是通过揭示算法启发式来研究内部工作原理，而算法启发式在赋予经过训练的模型具有出色的泛化特性方面可能发挥着同样重要的作用。鉴于深度学习的广泛适用性以及拟议研究中理论分析和实证研究的互补性，该项目特别适合将研究整合到教育和推广中。拟议的教育活动包括课程开发、暑期实习、黑客马拉松以及通过巴尔的摩当地项目进行教师外展。该项目研究了早期停止、批量归一化和 dropout 形式的显式算法正则化的作用，以及优化算法和网络架构的选择，以提供有助于泛化的足够的归纳偏差。该项目的第二个总体目标是更广泛地理解深度学习中的泛化现象。它旨在了解为什么记忆训练数据的系统仍然可以很好地泛化，神经网络架构如何实现迁移学习，以及如何设计强大的算法来保证深度学习解决方案在数据出现对抗性损坏的情况下也能泛化。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Robustness Guarantees for Adversarially Trained Neural Networks

对抗训练神经网络的鲁棒性保证

• 发表时间: 2024
• 期刊: Advances in neural information processing systems
• 作者: Poorya Mianjy, Raman Arora

Differentially Private Generalized Linear Models Revisited

重新审视差分私有广义线性模型

• 发表时间: 2022
• 期刊: Advances in neural information processing systems
• 作者: Raman Arora, Raef Bassily

FetchSGD: Communication-Efficient Federated Learning with Sketching

• 发表时间: 2020-07
• 期刊: ArXiv
• 作者: D. Rothchild;Ashwinee Panda;Enayat Ullah;Nikita Ivkin;I. Stoica;Vladimir Braverman;Joseph Gonzalez-Joseph-Gonzale

Generalization bounds for Kernel Canonical Correlation Analysis

核典型相关分析的泛化界限

• 发表时间: 2023
• 期刊: Transactions on machine learning research
• 作者: Enayat Ullah, Raman Arora

On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximation

基于线性函数逼近的离线强化学习的实例相关界限

• 发表时间: 2023
• 期刊: Proceedings of the AAAI Conference on Artificial Intelligence
• 作者: Nguyễn-Tang, Thanh;Yin, Ming;Gupta, Sunil;Venkatesh, Svetha;Arora, Raman
• 通讯作者: Arora, Raman