The Data-dependency Gap: A New Problem in the Learning Theory of Convolutional Neural Networks

在统计学习理论中，我们的目的是证明机器学习算法泛化能力的理论保证。该方法通常在于限定与算法相关联的函数类的复杂性。当复杂度较小时(与训练样本数相比)，算法具有较好的泛化能力。然而，对于神经网络来说，复杂性往往是非常大的。然而，神经网络--尤其是卷积神经网络--在广泛的应用中实现了前所未有的泛化。这种现象不能用标准的学习理论来解释。虽然大量文献通过分析训练过程中的内隐规则化提供了部分答案，但这一现象大体上还没有被很好地理解。在这个方案中，我们对神经网络“惊人的高”泛化能力引入了一个新的观点：数据依赖差距。我们认为，这些无法解释的概括能力的根本原因很可能在于数据本身的结构。我们的中心假设是，这些数据在神经网络训练中扮演着正则化的角色。这项提议的目的是验证这一假设。我们将进行实证评估并发展学习理论，根据数据中的结构以学习界限的形式存在。在这里，我们将把训练的CNN的权重与手头的观测输入联系起来，同时考虑到底层数据分布中的结构。我们专注于卷积神经网络，这是可以说是最突出的一类实用神经网络。然而，目前的工作可能会为分析其他类别的网络铺平道路(这可能会在战略计划的第二个资助期解决)。

In Statistical learning theory, we aim to prove theoretical guarantees on the generalization ability of machine learning algorithms. The approach usually consists in bounding the complexity of the function class associated with the algorithm. When the complexity is small (compared to the number of training samples), the algorithm is guaranteed to generalize well. For neural networks however, the complexity is oftentimes extremely large. Nevertheless, neural networks—and convolutional neural networks especially—have achieved unprecedented generalization in a wide range of applications. This phenomenon cannot be explained by standard learning theory. Although a rich body of literature provides partial answers through analysis of the implicit regularization imposed by the training procedure, the phenomenon is by large not well understood. In this proposal, we introduce a new viewpoint on the “surprisingly high” generalization ability of neural networks: the data-dependency gap. We argue that the fundamental reason for these unexplained generalization abilities may well lie in the structure of the data itself. Our central hypothesis is that the data acts as a regularizer on neural network training. The aim of this proposal is to verify this hypothesis. We will carry out empirical evaluations and develop learning theory, in the form of learning bounds depending on the structure in the data. Here we will connect the weights of trained CNNs with the observed inputs at hand, taking into account the structure in the underlying data distribution. We focus on convolutional neural networks, the arguably most prominent class of practical neural networks. However, the present work may pave the way for the analysis of other classes of networks (this may be tackled in the second funding period of the SPP).