Learning in the machine: To share or not to share?
Learning in the machine: To share or not to share?
复制标题
机器学习：分享还是不分享？
DOI：
10.1016/j.neunet.2020.03.016
复制
发表时间：
2020
期刊：
影响因子：
7.8
通讯作者：
Baldi, Pierre
中科院分区：
文献类型：
--
作者：
Ott, Jordan;Linstead, Erik;LaHaye, Nicholas;Baldi, Pierre
权重共享是卷积神经网络及其成功背后的支柱之一。然而，在大脑等物理神经系统中，体重分担是不可信的。这一差异提出了一个根本问题，即是否有必要分担权重。如果是，精确到什么程度？如果没有，有什么替代办法？本研究的目的是调查这些问题，主要是通过模拟的重量共享的假设是放松。从神经电路中获得灵感，我们探索了自由卷积网络和具有可变连接模式的神经元的使用。使用自由卷积网络，我们表明，虽然权重共享是一种实用的优化方法，但在计算机视觉应用中并不是必需的。此外，当使用正确翻译的数据（类似于视频）进行训练时，自由卷积网络与标准架构中观察到的性能相匹配。在假设数据是随机增强的情况下，自由卷积网络学习随机不变的表示，从而产生近似形式的权重共享。
Weight-sharing is one of the pillars behind Convolutional Neural Networks and their successes. However, in physical neural systems such as the brain, weight-sharing is implausible. This discrepancy raises the fundamental question of whether weight-sharing is necessary. If so, to which degree of precision? If not, what are the alternatives? The goal of this study is to investigate these questions, primarily through simulations where the weight-sharing assumption is relaxed. Taking inspiration from neural circuitry, we explore the use of Free Convolutional Networks and neurons with variable connection patterns. Using Free Convolutional Networks, we show that while weight-sharing is a pragmatic optimization approach, it is not a necessity in computer vision applications. Furthermore, Free Convolutional Networks match the performance observed in standard architectures when trained using properly translated data (akin to video). Under the assumption of translationally augmented data, Free Convolutional Networks learn translationally invariant representations that yield an approximate form of weight-sharing.