Collaborative Research: Foundations of Deep Learning: Theory, Robustness, and the Brain​

一个真正全面的机器学习理论，有可能像麦克斯韦方程那样，以同样深刻的方式为科学和工程提供信息。正是麦克斯韦的这一理论的发展，真正释放了电力的潜力，导致了无线电、雷达、计算机和互联网的出现。打个比方，深度学习（DL）在过去十年中已经找到了许多应用，到目前为止还没有一个全面的理论。一个最终的学习理论可以解释深度网络为什么和如何工作，以及它们的局限性是什么，因此可能会使更强大的学习方法的发展成为可能——特别是如果将深度学习与大脑研究重新联系起来的目标取得成果的话。从长远来看，开发和制造更好的智能机器的能力对任何技术型经济都至关重要。毕竟，即使在目前——仍然非常不完善——的状态下，深度学习正在或即将影响我们社会和生活的方方面面。研究人员还计划通过培训机器学习及其理论基础领域的数学、计算机科学、统计学、电子工程和计算神经科学等不同领域的年轻研究人员的教育目标来补充他们的理论研究。研究人员建议联合起来，通过他们的经验、专业知识、想法和洞察力，对深度学习的深奥奥秘进行多管齐下的合作攻击。研究目标有三个：发展对深度学习的良好的基础/数学理解；这样做是为了更普遍地推进对学习的基本理解；并通过解决DL的上述缺点来推进DL的实践。在六个基本要点中，前两个重点放在近似和样本（或估计）误差的预测误差的标准分解上。他们的目标是将逼近理论和可学习性理论中的经典结果扩展到深度学习。然后，这两种方法得到了一个专门针对深度学习的研究项目的支持：分析训练网络时梯度下降的动态。第四个主题是关于对对手和变化的鲁棒性，这是对理论的有力测试，对学习系统的实际部署也很重要。第五个重点是通过深度学习发展控制理论，以及探索深度强化学习的动态系统方面。最后一个主题将深度学习的研究与它的起源联系起来——也可能是它的未来：大脑中的神经元网络。提出的研究也有望推进学习理论的基础。这个项目的成功将为机器学习带来更清晰的数学技术，并为机器学习的鲁棒性奠定全面的基础。它还将最终推动超越深度学习的学习算法的发展，并为创造更智能的机器指引道路，并为我们自己的智能提供新的视角。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

A truly comprehensive theory of machine learning has the potential of informing science and engineering in the same profound way Maxwell’s equations did. It was the development of that theory by Maxwell that truly unleashed the potential of electricity, leading to radio, radars, computers, and the Internet. In an analogy, deep learning (DL) has found over the past decade many applications, so far without a comprehensive theory. An eventual theory of learning that explains why and how deep networks work and what their limitations are may thus enable the development of even more powerful learning approaches – especially if the goal of reconnecting DL to brain research bears fruit. In the long term, the ability to develop and build better intelligent machines will be essential to any technology-based economy. After all, even in its current – still highly imperfect –state, DL is impacting or about to impact just about every aspect of our society and life. The investigators also plan to complement their theoretical research with the educational goal of training a diverse population of young researchers from mathematics, computer science, statistics, electrical engineering, and computational neuroscience in the field of machine learning and of its theoretical underpinnings.The investigators propose to join forces in a multi-pronged and collaborative assault on the profound mysteries of DL, informed by the sum of their experience, expertise, ideas, and insight. The research goals are threefold: to develop a sound foundational/mathematical understanding of DL; in doing so to advance the foundational understanding of learning more generally; and to advance the practice of DL by addressing its above-mentioned weaknesses. Of six foundational thrusts, the first two focus on the standard decomposition of the prediction error in approximation and sample (or estimation) error. Their goal is to extend classical results in approximation theory and theory of learnability to DL. These two are then supported by a research project that is specific to deep learning: analysis of the dynamics of gradient descent in training a network. The fourth theme is about robustness against adversaries and shifts, a powerful test for theories which is also important for practical deployment of learning systems. The fifth thrust is about developing the theory of control through DL, as well as exploring dynamical systems aspects of deep reinforcement learning. The final topic connects research on DL to its origins - and possibly its future: networks of neurons in the brain. The proposed research also promises to advance the foundations of learning theory. Success in this project will result in sharper mathematical techniques for machine learning and comprehensive foundations of machine learning robustness, broadly construed. It will also ultimately enable development of learning algorithms that transcend deep learning and guide the way towards creating more intelligent machines, and shed new light on our own intelligence.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.