The Data-dependency Gap: A New Problem in the Learning Theory of Convolutional Neural Networks

数据依赖性差距：卷积神经网络学习理论的新问题

• 批准号: 464252197
• 负责人: Professor Dr. Marius Kloft
• 金额: --
• 依托单位: Arbeitsgruppe Maschinelles Lernen
• 依托单位国家: 德国
• 项目类别: Priority Programmes
• 资助国家: 德国
• 项目状态: 未结题
• 来源: https://gepris.dfg.de/gepris/projekt/464252197?language=en
• 关键词: Data; dependency; Gap; New; Problem

In Statistical learning theory, we aim to prove theoretical guarantees on the generalization ability of machine learning algorithms. The approach usually consists in bounding the complexity of the function class associated with the algorithm. When the complexity is small (compared to the number of training samples), the algorithm is guaranteed to generalize well. For neural networks however, the complexity is oftentimes extremely large. Nevertheless, neural networks—and convolutional neural networks especially—have achieved unprecedented generalization in a wide range of applications. This phenomenon cannot be explained by standard learning theory. Although a rich body of literature provides partial answers through analysis of the implicit regularization imposed by the training procedure, the phenomenon is by large not well understood. In this proposal, we introduce a new viewpoint on the “surprisingly high” generalization ability of neural networks: the data-dependency gap. We argue that the fundamental reason for these unexplained generalization abilities may well lie in the structure of the data itself. Our central hypothesis is that the data acts as a regularizer on neural network training. The aim of this proposal is to verify this hypothesis. We will carry out empirical evaluations and develop learning theory, in the form of learning bounds depending on the structure in the data. Here we will connect the weights of trained CNNs with the observed inputs at hand, taking into account the structure in the underlying data distribution. We focus on convolutional neural networks, the arguably most prominent class of practical neural networks. However, the present work may pave the way for the analysis of other classes of networks (this may be tackled in the second funding period of the SPP).

在统计学习理论中，我们的目标是证明机器学习算法泛化能力的理论保证。该方法通常包括限制与算法相关联的函数类的复杂性。当复杂度较小时（与训练样本的数量相比），该算法可以保证很好的推广。然而，对于神经网络来说，复杂性往往非常大。然而，神经网络，特别是卷积神经网络，在广泛的应用中取得了前所未有的普遍性。这种现象无法用标准学习理论来解释。虽然丰富的文献提供了部分答案，通过分析所施加的训练过程中的隐式正则化，这一现象是由大不好理解。在这个提议中，我们引入了一个关于神经网络“令人惊讶的高”泛化能力的新观点：数据依赖差距。我们认为，这些无法解释的泛化能力的根本原因可能在于数据本身的结构。我们的中心假设是，数据在神经网络训练中充当正则化器。本提案的目的是验证这一假设。我们将进行实证评估，并发展学习理论，根据数据中的结构学习边界的形式。在这里，我们将把训练好的CNN的权重与观察到的输入连接起来，同时考虑到底层数据分布的结构。我们专注于卷积神经网络，可以说是最突出的实用神经网络类别。然而，目前的工作可能为分析其他类别的网络铺平道路（这可能在SPP的第二个供资期内解决）。