Towards Principled Design of Deep Convolutional Networks: Introducing SimpNet
Towards Principled Design of Deep Convolutional Networks: Introducing SimpNet
复制标题
DOI：
--
复制
发表时间：
2018-02
期刊：
影响因子：
--
通讯作者：
S. H. HasanPour;Mohammad Rouhani;Mohsen Fayyaz-;M. Sabokrou;E. Adeli
中科院分区：
文献类型：
--
作者：
S. H. HasanPour;Mohammad Rouhani;Mohsen Fayyaz-;M. Sabokrou;E. Adeli
主要获奖的卷积神经网络（CNN），如VGGNet，ResNet，DenseNet等，包括数千万到数亿个参数，这会带来相当大的计算和内存开销。这限制了它们在训练和优化现实世界应用程序中的实际使用。相反，轻量级架构，如SqueezeNet，正在被提出来解决这个问题。然而，它们主要遭受低精度，因为它们在处理能力和效率之间折衷。这些低效率主要源于遵循一个特设的设计程序。在这项工作中，我们讨论并提出了几个关键的设计原则，一个有效的架构设计和详细的直觉有关的设计过程的不同方面。此外，我们引入了一个新的层，称为{\it SAF-pooling}，以提高网络的泛化能力，同时通过选择最佳特征来保持简单。基于这些原则，我们提出了一个名为{\it SimpNet}的简单架构。我们的经验表明，CNONet提供了一个很好的权衡之间的计算/内存效率和准确性仅仅基于这些原始的，但至关重要的原则。在几个著名的基准测试中，RISNet的性能优于更深入、更复杂的架构，如VGGNet、ResNet、WideResidualNet等，同时参数和操作的数量减少了2到25倍。我们在标准数据集（如CIFAR 10，CIFAR 100，MNIST和SVHN）上获得了最先进的结果（在准确性和所涉及的参数数量之间的平衡方面）。这些实现可从\href{url}{this https URL}获得。
Major winning Convolutional Neural Networks (CNNs), such as VGGNet, ResNet, DenseNet, \etc, include tens to hundreds of millions of parameters, which impose considerable computation and memory overheads. This limits their practical usage in training and optimizing for real-world applications. On the contrary, light-weight architectures, such as SqueezeNet, are being proposed to address this issue. However, they mainly suffer from low accuracy, as they have compromised between the processing power and efficiency. These inefficiencies mostly stem from following an ad-hoc designing procedure. In this work, we discuss and propose several crucial design principles for an efficient architecture design and elaborate intuitions concerning different aspects of the design procedure. Furthermore, we introduce a new layer called {\it SAF-pooling} to improve the generalization power of the network while keeping it simple by choosing best features. Based on such principles, we propose a simple architecture called {\it SimpNet}. We empirically show that SimpNet provides a good trade-off between the computation/memory efficiency and the accuracy solely based on these primitive but crucial principles. SimpNet outperforms the deeper and more complex architectures such as VGGNet, ResNet, WideResidualNet \etc, on several well-known benchmarks, while having 2 to 25 times fewer number of parameters and operations. We obtain state-of-the-art results (in terms of a balance between the accuracy and the number of involved parameters) on standard datasets, such as CIFAR10, CIFAR100, MNIST and SVHN. The implementations are available at \href{url}{this https URL}.