Collaborative Research: Foundations of Deep Learning: Theory, Robustness, and the Brain​

协作研究：深度学习的基础：理论、稳健性和大脑 —

• 批准号: 2134059
• 负责人: Christos Papadimitriou
• 金额: $ 30万
• 依托单位: Columbia University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-12-01 至 2024-11-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2134059&HistoricalAwards=false
• 关键词: Collaborative; Research; Foundations; Deep; Learning

A truly comprehensive theory of machine learning has the potential of informing science and engineering in the same profound way Maxwell’s equations did. It was the development of that theory by Maxwell that truly unleashed the potential of electricity, leading to radio, radars, computers, and the Internet. In an analogy, deep learning (DL) has found over the past decade many applications, so far without a comprehensive theory. An eventual theory of learning that explains why and how deep networks work and what their limitations are may thus enable the development of even more powerful learning approaches – especially if the goal of reconnecting DL to brain research bears fruit. In the long term, the ability to develop and build better intelligent machines will be essential to any technology-based economy. After all, even in its current – still highly imperfect –state, DL is impacting or about to impact just about every aspect of our society and life. The investigators also plan to complement their theoretical research with the educational goal of training a diverse population of young researchers from mathematics, computer science, statistics, electrical engineering, and computational neuroscience in the field of machine learning and of its theoretical underpinnings.The investigators propose to join forces in a multi-pronged and collaborative assault on the profound mysteries of DL, informed by the sum of their experience, expertise, ideas, and insight. The research goals are threefold: to develop a sound foundational/mathematical understanding of DL; in doing so to advance the foundational understanding of learning more generally; and to advance the practice of DL by addressing its above-mentioned weaknesses. Of six foundational thrusts, the first two focus on the standard decomposition of the prediction error in approximation and sample (or estimation) error. Their goal is to extend classical results in approximation theory and theory of learnability to DL. These two are then supported by a research project that is specific to deep learning: analysis of the dynamics of gradient descent in training a network. The fourth theme is about robustness against adversaries and shifts, a powerful test for theories which is also important for practical deployment of learning systems. The fifth thrust is about developing the theory of control through DL, as well as exploring dynamical systems aspects of deep reinforcement learning. The final topic connects research on DL to its origins - and possibly its future: networks of neurons in the brain. The proposed research also promises to advance the foundations of learning theory. Success in this project will result in sharper mathematical techniques for machine learning and comprehensive foundations of machine learning robustness, broadly construed. It will also ultimately enable development of learning algorithms that transcend deep learning and guide the way towards creating more intelligent machines, and shed new light on our own intelligence.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

一个真正全面的机器学习理论有可能以麦克斯韦方程组同样深刻的方式为科学和工程提供信息。正是麦克斯韦对这一理论的发展真正释放了电力的潜力，导致了无线电、雷达、计算机和互联网的出现。打个比方，深度学习（DL）在过去十年中发现了许多应用，但到目前为止还没有一个全面的理论。因此，一个解释深度网络为什么和如何工作以及它们的局限性的最终学习理论可能会使更强大的学习方法的发展成为可能-特别是如果将深度学习与大脑研究重新联系起来的目标取得成果的话。从长远来看，开发和制造更好的智能机器的能力对任何以技术为基础的经济都至关重要。毕竟，即使是在它目前仍然非常不完善的状态下，深度学习正在或即将影响我们社会和生活的方方面面。研究人员还计划在理论研究的基础上，培养一批来自数学、计算机科学、统计学、电气工程和计算神经科学等领域的年轻研究人员，以实现机器学习及其理论基础的教育目标。研究人员建议联合起来，多管齐下，协同攻击深度学习的奥秘，通过他们的经验，专业知识，想法和洞察力的总和来了解。研究目标有三个方面：发展一个健全的基础/数学的DL理解;这样做，以推进学习的基本理解更普遍;并通过解决其上述弱点，以推进DL的实践。在六个基本要点中，前两个重点是近似和样本（或估计）误差中预测误差的标准分解。他们的目标是将近似理论和可学习性理论中的经典结果推广到DL。然后，这两项研究得到了一个专门针对深度学习的研究项目的支持：分析训练网络中梯度下降的动态。第四个主题是关于对对手和变化的鲁棒性，这是对理论的有力检验，对学习系统的实际部署也很重要。第五个重点是通过深度学习发展控制理论，以及探索深度强化学习的动力系统方面。最后一个主题将DL的研究与其起源-以及可能的未来联系起来：大脑中的神经元网络。拟议中的研究也有望推进学习理论的基础。该项目的成功将为机器学习带来更清晰的数学技术，并为机器学习的鲁棒性提供全面的基础。该奖项反映了NSF的法定使命，通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

A biologically plausible parser

生物学上合理的解析器

• 发表时间: 2021
• 期刊: Transactions of the Association for Computational Linguistics
• 影响因子: 10.9
• 作者: Daniel Mitropolsky, Michael Collins

Self-Attention Networks Can Process Bounded Hierarchical Languages

• DOI: 10.18653/v1/2021.acl-long.292
• 发表时间: 2021-05
• 作者: Shunyu Yao;Binghui Peng;C. Papadimitriou;Karthik Narasimhan

Conformal Sensitivity Analysis for Individual Treatment Effects

• DOI: 10.1080/01621459.2022.2102503
• 发表时间: 2021-12
• 期刊: Journal of the American Statistical Association
• 影响因子: 3.7
• 作者: Mingzhang Yin;Claudia Shi;Yixin Wang;D. Blei