A Fresh Look at our Understanding of Machine Learning

点击翻译按钮获取中文摘要

Machine Learning and other "Artificial Intelligence" technologies are rapidly being deployed across industry, science, and government. Much of this progress is due to the application of deep neural networks. Despite empirical success stories, our theoretical understanding of neural networks is still very limited, even though neural networks have been studied for decades. One reason is that modern neural networks are much larger and deeper. Another is that the way we train neural networks on data has evolved. While we are still relatively in the dark, a number of empirical phenomena can serve as beacons. One such phenomenon is interpolation, where neural networks can be trained to perform perfectly on training data, even if the training data in corrupted by noise. Remarkably, neural network classifiers do not seem to suffer from overfitting in this regime. Devising an explanation for this phenomenon is a major open problem. Another phenomenon relates to the role of data in generalization performance. Why does the standard learning algorithm, stochastic gradient descent, learn an accurate classifier on real data, when the same algorithm, running on the same architecture, overfits badly on corrupted data? What property of the data explains this? And can we predict it? As machine learning expands into sensitive application areas such as healthcare, transportation, and policy making, it is imperative that we develop better understanding. The central goal of my research program is to bridge the gap between empirical and theoretical performance, building on the progress made within the statistical learning community, while scrutinizing those aspects of the foundation that may divide theory and practice. In addition to studying this gap theoretically, this research program aims to use empirical methods to understand the limitations of existing theory and also inspire and evaluate new theory.