A Fresh Look at our Understanding of Machine Learning

机器学习和其他“人工智能”技术正迅速在工业、科学和政府部门部署。这一进步在很大程度上要归功于深度神经网络的应用。尽管有经验上的成功案例，但我们对神经网络的理论理解仍然非常有限，尽管神经网络已经研究了几十年。其中一个原因是现代神经网络的规模和深度都要大得多。另一个原因是，我们训练神经网络处理数据的方式已经发生了变化。虽然我们仍然处于相对黑暗的状态，但许多经验现象可以充当灯塔。一种这样的现象是内插，其中神经网络可以被训练成在训练数据上完美地执行，即使训练数据受到噪声的破坏。值得注意的是，神经网络分类器似乎并没有在这种制度下受到过度拟合的影响。如何解释这一现象是一个重大的悬而未决的问题。另一个现象与数据在推广工作中的作用有关。为什么标准的学习算法，随机梯度下降，在真实数据上学习一个准确的分类器，而同样的算法，运行在相同的架构上，严重不适合被破坏的数据？数据的什么属性可以解释这一点？我们能预测到吗？ 随着机器学习扩展到医疗、交通和政策制定等敏感应用领域，我们迫切需要更好地理解。我的研究计划的中心目标是在统计学习社区内取得的进展的基础上，弥合经验和理论表现之间的差距，同时仔细审查可能导致理论和实践分裂的基础方面。除了从理论上研究这一差距，本研究计划旨在用实证的方法来理解现有理论的局限性，并启发和评价新的理论。

Machine Learning and other ”Artificial Intelligence” technologies are rapidly being deployed across industry, science, and government. Much of this progress is due to the application of deep neural networks. Despite empirical success stories, our theoretical understanding of neural networks is still very limited, even though neural networks have been studied for decades. One reason is that modern neural networks are much larger and deeper. Another is that the way we train neural networks on data has evolved. While we are still relatively in the dark, a number of empirical phenomena can serve as beacons. One such phenomenon is interpolation, where neural networks can be trained to perform perfectly on training data, even if the training data in corrupted by noise. Remarkably, neural network classifiers do not seem to suffer from overfitting in this regime. Devising an explanation for this phenomenon is a major open problem. Another phenomenon relates to the role of data in generalization performance. Why does the standard learning algorithm, stochastic gradient descent, learn an accurate classifier on real data, when the same algorithm, running on the same architecture, overfits badly on corrupted data? What property of the data explains this? And can we predict it? As machine learning expands into sensitive application areas such as healthcare, transportation, and policy making, it is imperative that we develop better understanding. The central goal of my research program is to bridge the gap between empirical and theoretical performance, building on the progress made within the statistical learning community, while scrutinizing those aspects of the foundation that may divide theory and practice. In addition to studying this gap theoretically, this research program aims to use empirical methods to understand the limitations of existing theory and also inspire and evaluate new theory.