Principled approaches to deep learning: generalization under distribution shift and predictive uncertainty

深度学习（DL）解决了以前使用传统机器学习（ML）方法无法解决的问题。然而，与具有坚实理论基础的ML不同，DL迄今为止依赖于工程实践。特别是，虽然深度学习在计算机应用中非常成功，但到目前为止，它在现实世界问题（例如自动驾驶汽车）中的应用仍然有限。为了继续扩大应用，深度学习需要一个理论基础，以误差估计的形式，它提供了对新输入模型准确性的控制。30年前，机器学习（ML）面临的情况与DL目前面临的情况类似：方法远远领先于理论。在短时间内，机器学习理论能够解决理论问题，使机器学习模型能够安全地部署在广泛的应用中。所提出的程序将为深度学习做30年前为ML所做的事情。误差估计可以在两种设置中解决。第一个是泛化误差边界，它在模型看到输入之前应用。第二个是预测的不确定性，它在模型看到输入之后（但在做出决定之前）应用。例如，假设我们有一个决策问题，我们只能在错误概率小于1%的情况下采取行动，但我们的模型误差平均为5%。根据具体情况，预测的不确定性可以告诉我们，根据哪些输入，模型的精度足够高，可以采取行动。对于第二个例子，我们使用图像数据库训练计算机视觉模型，但我们希望将它们部署在真实世界的图像上，这些图像在统计上与训练集不同。我们建议扩展数据转换技术和测量数据集差异的方法，以便更好地估计真实世界图像上的模型误差。该提案将通过建立DL out-of-distribution （OOD）泛化的数学理论来解决在更广泛的环境中应用深度学习的问题。会的。制定合适的定义和假设，使我们能够用数学术语来描述深度学习的分布外泛化问题。2.根据相关的问题输入，证明一个估计（高概率）OOD泛化差距的定理。3.通过数学建模确定相关的问题输入。确定相关的假设需要应用科学的方法，在这种情况下，对应于设计用于探测深度神经网络泛化行为的计算机实验。精确地陈述定义和假设涉及数学建模。证明上述定理涉及数学分析。

Deep Learning (DL) addresses problems which were previously not possible using traditional Machine Learning (ML) methods. However, unlike ML, which has a solid theoretical foundation, DL has so far relied engineering practices. In particular, while very successful in computer applications, DL has so far been limited in its applications to real world problems, for example autonomous vehicles. In order to continue to broaden the applications, deep learning requires a theoretical foundation, in the form of error estimates, which provide control over the accuracy of models on new inputs. Thirty years ago, Machine Learning (ML) was in a situation similar to the one currently faced by DL: the methods were far ahead of the theory. In a short time, ML theory was able to solve theoretical problems, allowing ML models to be safely deployed in a wide range of applications. The proposed program will do for DL what was done thirty years ago for ML. Error estimation can be addressed in two settings. The first is generalization error bounds, which apply before the inputs are seen by the model. The second is predictive uncertainty, which applies after the inputs are seen by the model (but before a decision is made). For example, suppose we have a decision problem where we can only act if the probability of error is less than 1%, but our model error is 5%, on average. Predictive uncertainty can tell us, on a case by case basis, on which inputs the model accuracy is high enough to act. For a second example, we train computer vision models using databases of images, but we want to deploy them on real world images, which are statistically different from the training set. We propose to extend data transformation techniques, and methods for measuring dataset differences, in order to better estimate the model error on real world images. This proposal will address the problem of applying deep learning in a broader setting by building a mathematical theory for DL out-of-distribution (OOD) generalization. It will 1.Formulate suitable definitions and assumptions which allow us to state the deep learning out-of- distribution generalization problem in mathematical terms. 2.Prove a theorem estimating (with high probability) the OOD generalization gap, in terms of relevant problem inputs. 3.Determine relevant problem inputs through mathematical modelling. Determining the relevant hypotheses requires applying the scientific method, which, in this case corresponds to computer experiments designed to probe the generalization behaviour of deep neural networks. Stating the definitions and assumptions precisely involves mathematical modelling. Proving the theorem described above involves mathematical analysis.