Empirical Study of Easy and Hard Examples in CNN Training
Empirical Study of Easy and Hard Examples in CNN Training
复制标题
CNN 训练中简单和困难示例的实证研究
DOI：
10.1007/978-3-030-36808-1_20
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
Hideki Nakayama
中科院分区：
文献类型：
--
作者：
Ikki Kishida;Hideki Nakayama
深度神经网络（DNN）具有很好的泛化能力，尽管它们的规模很大，并且能够记住所有的例子。有一种假设是DNN从简单模式开始学习，并且该假设是基于在早期训练阶段（即，简单的例子）和错误分类的例子（即，硬例子）。简单的例子是DNN从特定模式开始学习的证据，并且有一个一致的学习过程。重要的是要知道DNN如何学习模式并获得泛化能力，然而，简单和困难示例的属性没有得到彻底的研究（例如，对泛化和视觉外观的贡献）。在这项工作中，我们分别研究了不同卷积神经网络（CNN）架构的简单和困难示例的相似性，评估这些示例如何有助于泛化。我们的研究结果表明，简单的例子在视觉上彼此相似，而困难的例子在视觉上是多样的，这两个例子在很大程度上在不同的CNN架构中共享。此外，虽然困难的例子往往比简单的例子更有助于泛化，但删除大量简单的例子会导致泛化能力差。通过分析这些结果，我们假设数据集中的偏差和随机梯度下降（SGD）是CNN具有一致的简单和困难示例的原因。此外，我们表明，大规模的分类数据集可以有效地压缩，通过使用在这项工作中提出的easiness。
Deep Neural Networks (DNNs) generalize well despite their massive size and capability of memorizing all examples. There is a hypothesis that DNNs start learning from simple patterns and the hypothesis is based on the existence of examples that are consistently well-classified at the early training stage (i.e.,easy examples) and examples misclassified (i.e.,hard examples). Easy examples are the evidence that DNNs start learning from specific patterns and there is a consistent learning process. It is important to know how DNNs learn patterns and obtain generalization ability, however, properties of easy and hard examples are not thoroughly investigated (e.g., contributions to generalization and visual appearances). In this work, we study the similarities of easy and hard examples respectively for different Convolutional Neural Network (CNN) architectures, assessing how those examples contribute to generalization. Our results show that easy examples are visually similar to each other and hard examples are visually diverse, and both examples are largely shared across different CNN architectures. Moreover, while hard examples tend to contribute more to generalization than easy examples, removing a large number of easy examples leads to poor generalization. By analyzing those results, we hypothesize that biases in a dataset and Stochastic Gradient Descent (SGD) are the reasons why CNNs have consistent easy and hard examples. Furthermore, we show that large scale classification datasets can be efficiently compressed by usingeasinessproposed in this work.