Collaborative Research: Foundations of Deep Learning: Theory, Robustness, and the Brain​

一个真正全面的机器学习理论有可能像麦克斯韦方程一样深刻地为科学和工程提供信息。正是麦克斯韦对这一理论的发展，真正释放了电的潜力，导致了无线电、雷达、计算机和互联网的出现。打个比方，深度学习在过去十年里发现了许多应用，但到目前为止还没有一个全面的理论。一种最终的学习理论解释了深度网络为什么以及如何工作，以及它们的局限性是什么，因此可能会开发出更强大的学习方法--特别是如果重新将数字学习与大脑研究联系起来的目标取得成果的话。从长远来看，开发和制造更好的智能机器的能力将对任何以技术为基础的经济至关重要。毕竟，即使在目前--仍然非常不完美--的状态下，数字图书馆正在或即将影响我们社会和生活的方方面面。研究人员还计划用教育目标来补充他们的理论研究，即在机器学习及其理论基础领域培养一批来自数学、计算机科学、统计学、电气工程和计算神经科学的年轻研究人员。调查人员建议联合起来，以他们的经验、专业知识、想法和洞察力的总和为基础，多管齐下地合作攻破数字图书馆的深刻奥秘。研究的目标有三个：建立对数字学习的基础/数学理解；通过这样做来促进对学习的更广泛的基础理解；通过解决上述弱点来促进数字学习的实践。在六个基本假设中，前两个集中于近似预测误差和样本(或估计)误差的标准分解。他们的目标是将逼近理论和可学习性理论中的经典结果推广到数字图书馆。这两点随后得到了一个专门针对深度学习的研究项目的支持：分析网络训练中梯度下降的动态。第四个主题是关于对对手和变化的稳健性，这是对理论的强大考验，对学习系统的实际部署也很重要。第五个重点是关于通过动态学习发展控制理论，以及探索深度强化学习的动力系统方面。最后一个主题将对DL的研究与它的起源--可能还有它的未来--联系在一起：大脑中的神经元网络。所提出的研究也有望推进学习理论的基础。这个项目的成功将带来更敏锐的机器学习数学技术和广泛解释的机器学习稳健性的全面基础。它还将最终使学习算法的开发超越深度学习，并指导创造更智能的机器，并揭示我们自己的智能。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

A truly comprehensive theory of machine learning has the potential of informing science and engineering in the same profound way Maxwell’s equations did. It was the development of that theory by Maxwell that truly unleashed the potential of electricity, leading to radio, radars, computers, and the Internet. In an analogy, deep learning (DL) has found over the past decade many applications, so far without a comprehensive theory. An eventual theory of learning that explains why and how deep networks work and what their limitations are may thus enable the development of even more powerful learning approaches – especially if the goal of reconnecting DL to brain research bears fruit. In the long term, the ability to develop and build better intelligent machines will be essential to any technology-based economy. After all, even in its current – still highly imperfect –state, DL is impacting or about to impact just about every aspect of our society and life. The investigators also plan to complement their theoretical research with the educational goal of training a diverse population of young researchers from mathematics, computer science, statistics, electrical engineering, and computational neuroscience in the field of machine learning and of its theoretical underpinnings.The investigators propose to join forces in a multi-pronged and collaborative assault on the profound mysteries of DL, informed by the sum of their experience, expertise, ideas, and insight. The research goals are threefold: to develop a sound foundational/mathematical understanding of DL; in doing so to advance the foundational understanding of learning more generally; and to advance the practice of DL by addressing its above-mentioned weaknesses. Of six foundational thrusts, the first two focus on the standard decomposition of the prediction error in approximation and sample (or estimation) error. Their goal is to extend classical results in approximation theory and theory of learnability to DL. These two are then supported by a research project that is specific to deep learning: analysis of the dynamics of gradient descent in training a network. The fourth theme is about robustness against adversaries and shifts, a powerful test for theories which is also important for practical deployment of learning systems. The fifth thrust is about developing the theory of control through DL, as well as exploring dynamical systems aspects of deep reinforcement learning. The final topic connects research on DL to its origins - and possibly its future: networks of neurons in the brain. The proposed research also promises to advance the foundations of learning theory. Success in this project will result in sharper mathematical techniques for machine learning and comprehensive foundations of machine learning robustness, broadly construed. It will also ultimately enable development of learning algorithms that transcend deep learning and guide the way towards creating more intelligent machines, and shed new light on our own intelligence.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.