Diving deeper into mentee networks
Diving deeper into mentee networks
复制标题
DOI：
--
复制
发表时间：
2016-04
期刊：
影响因子：
--
通讯作者：
Ragav Venkatesan;Baoxin Li
中科院分区：
文献类型：
--
作者：
Ragav Venkatesan;Baoxin Li
现代计算机视觉完全是关于拥有强大的图像表示。越来越深的卷积神经网络已经使用越来越大的数据集建立起来，并公开可用。一大批计算机视觉科学家使用这些预先训练的网络，在各种任务中取得了不同程度的成功。尽管在复制这些网络方面取得了巨大的成功，但表征空间并不是以传统的方式从目标数据集中学习的。选择使用预先训练的网络而不是从零开始学习的网络的原因之一是，较小的数据集提供的监管较少，需要细致的正规化、较小和谨慎的学习率调整，甚至可以实现稳定的学习而不会出现权重爆炸。通常情况下，大型深层网络不可移植，这就需要从头开始学习中型网络的能力。在本文中，我们将通过从大型预先训练的网络获得额外的监督，从零开始对这些中型网络进行更深入的培训。与使用L2、L1和辍学等常见正则化技术训练的网络相比，这种学习还提供了更好的泛化精度。我们表明，通过这种方式学习的特征比那些独立学习的特征更普遍。我们研究了这类网络的各种特征，发现了一些有趣的行为。
Modern computer vision is all about the possession of powerful image representations. Deeper and deeper convolutional neural networks have been built using larger and larger datasets and are made publicly available. A large swath of computer vision scientists use these pre-trained networks with varying degrees of successes in various tasks. Even though there is tremendous success in copying these networks, the representational space is not learnt from the target dataset in a traditional manner. One of the reasons for opting to use a pre-trained network over a network learnt from scratch is that small datasets provide less supervision and require meticulous regularization, smaller and careful tweaking of learning rates to even achieve stable learning without weight explosion. It is often the case that large deep networks are not portable, which necessitates the ability to learn mid-sized networks from scratch. In this article, we dive deeper into training these mid-sized networks on small datasets from scratch by drawing additional supervision from a large pre-trained network. Such learning also provides better generalization accuracies than networks trained with common regularization techniques such as l2, l1 and dropouts. We show that features learnt thus, are more general than those learnt independently. We studied various characteristics of such networks and found some interesting behaviors.