Principled approaches to deep learning: generalization under distribution shift and predictive uncertainty

深度学习(DL)解决了以前使用传统机器学习(ML)方法无法解决的问题。然而，与拥有坚实理论基础的ML不同，DL到目前为止一直依赖于工程实践。特别是，虽然在计算机应用方面非常成功，但到目前为止，DL在现实世界问题上的应用一直受到限制，例如自动驾驶汽车。为了继续扩大应用，深度学习需要以误差估计的形式提供理论基础，以控制关于新投入的模型的准确性。30年前，机器学习(ML)的情况与数字图书馆目前所面临的情况类似：方法远远领先于理论。在很短的时间内，ML理论能够解决理论问题，使ML模型能够安全地部署在广泛的应用中。拟议的计划将为DL做30年前为ML所做的事情。误差估计可以在两种设置中解决。第一个是泛化误差界，在模型看到输入之前适用。第二种是预测不确定性，它适用于模型看到输入之后(但在做出决定之前)。例如，假设我们有一个决策问题，其中我们只有在错误概率小于1%时才能采取行动，但我们的模型误差平均为5%。预测的不确定性可以告诉我们，在个案的基础上，模型的输入精度足够高，可以采取行动。对于第二个例子，我们使用图像数据库来训练计算机视觉模型，但我们希望将它们部署在真实世界的图像上，这些图像在统计上与训练集不同。我们建议扩展数据转换技术和测量数据集差异的方法，以便更好地估计真实世界图像上的模型误差。这一建议将通过建立用于分布外(OOD)泛化的数学理论来解决在更广泛的环境中应用深度学习的问题。它将1.提出适当的定义和假设，使我们能够用数学术语来描述深度学习的非分布泛化问题。2.证明了一个关于相关问题输入的估计OOD泛化差距的(高概率)定理。3.通过数学建模确定相关的问题输入。确定相关假设需要应用科学方法，在这种情况下，这相当于为探索深层神经网络的泛化行为而设计的计算机实验。准确地说明定义和假设涉及到数学建模。证明上述定理需要进行数学分析。

Deep Learning (DL) addresses problems which were previously not possible using traditional Machine Learning (ML) methods. However, unlike ML, which has a solid theoretical foundation, DL has so far relied engineering practices. In particular, while very successful in computer applications, DL has so far been limited in its applications to real world problems, for example autonomous vehicles. In order to continue to broaden the applications, deep learning requires a theoretical foundation, in the form of error estimates, which provide control over the accuracy of models on new inputs. Thirty years ago, Machine Learning (ML) was in a situation similar to the one currently faced by DL: the methods were far ahead of the theory. In a short time, ML theory was able to solve theoretical problems, allowing ML models to be safely deployed in a wide range of applications. The proposed program will do for DL what was done thirty years ago for ML. Error estimation can be addressed in two settings. The first is generalization error bounds, which apply before the inputs are seen by the model. The second is predictive uncertainty, which applies after the inputs are seen by the model (but before a decision is made). For example, suppose we have a decision problem where we can only act if the probability of error is less than 1%, but our model error is 5%, on average. Predictive uncertainty can tell us, on a case by case basis, on which inputs the model accuracy is high enough to act. For a second example, we train computer vision models using databases of images, but we want to deploy them on real world images, which are statistically different from the training set. We propose to extend data transformation techniques, and methods for measuring dataset differences, in order to better estimate the model error on real world images. This proposal will address the problem of applying deep learning in a broader setting by building a mathematical theory for DL out-of-distribution (OOD) generalization. It will 1.Formulate suitable definitions and assumptions which allow us to state the deep learning out-of- distribution generalization problem in mathematical terms. 2.Prove a theorem estimating (with high probability) the OOD generalization gap, in terms of relevant problem inputs. 3.Determine relevant problem inputs through mathematical modelling. Determining the relevant hypotheses requires applying the scientific method, which, in this case corresponds to computer experiments designed to probe the generalization behaviour of deep neural networks. Stating the definitions and assumptions precisely involves mathematical modelling. Proving the theorem described above involves mathematical analysis.