A Fresh Look at our Understanding of Machine Learning

机器学习和其他“人工智能”技术正在工业、科学和政府中迅速部署。这一进展在很大程度上归功于深度神经网络的应用。尽管有经验上的成功案例，但我们对神经网络的理论理解仍然非常有限，尽管神经网络已经研究了几十年。原因之一是现代神经网络更大更深。另一个原因是，我们在数据上训练神经网络的方式已经发生了变化。虽然我们仍然相对处于黑暗之中，但一些经验现象可以作为信标。其中一种现象是插值，神经网络可以被训练成在训练数据上完美地执行，即使训练数据被噪声破坏。值得注意的是，神经网络分类器在这种情况下似乎不会受到过拟合的影响。对这种现象的解释是一个主要的悬而未决的问题。另一个现象与数据在泛化性能中的作用有关。为什么标准的学习算法，随机梯度下降，在真实的数据上学习一个准确的分类器，当同样的算法，在同样的架构上运行，在损坏的数据上过度拟合？数据的什么属性解释了这一点？我们能预测吗？ 随着机器学习扩展到医疗保健、交通和政策制定等敏感应用领域，我们必须更好地理解。我的研究计划的中心目标是弥合经验和理论性能之间的差距，建立在统计学习社区内取得的进展，同时仔细检查那些方面的基础，可能会分裂理论和实践。除了从理论上研究这一差距外，本研究计划还旨在使用实证方法来了解现有理论的局限性，并启发和评估新理论。

Machine Learning and other ”Artificial Intelligence” technologies are rapidly being deployed across industry, science, and government. Much of this progress is due to the application of deep neural networks. Despite empirical success stories, our theoretical understanding of neural networks is still very limited, even though neural networks have been studied for decades. One reason is that modern neural networks are much larger and deeper. Another is that the way we train neural networks on data has evolved. While we are still relatively in the dark, a number of empirical phenomena can serve as beacons. One such phenomenon is interpolation, where neural networks can be trained to perform perfectly on training data, even if the training data in corrupted by noise. Remarkably, neural network classifiers do not seem to suffer from overfitting in this regime. Devising an explanation for this phenomenon is a major open problem. Another phenomenon relates to the role of data in generalization performance. Why does the standard learning algorithm, stochastic gradient descent, learn an accurate classifier on real data, when the same algorithm, running on the same architecture, overfits badly on corrupted data? What property of the data explains this? And can we predict it? As machine learning expands into sensitive application areas such as healthcare, transportation, and policy making, it is imperative that we develop better understanding. The central goal of my research program is to bridge the gap between empirical and theoretical performance, building on the progress made within the statistical learning community, while scrutinizing those aspects of the foundation that may divide theory and practice. In addition to studying this gap theoretically, this research program aims to use empirical methods to understand the limitations of existing theory and also inspire and evaluate new theory.