Go with the flow: Adaptive control for Neural ODEs
Go with the flow: Adaptive control for Neural ODEs
复制标题
DOI：
--
复制
发表时间：
2020-06
期刊：
影响因子：
--
通讯作者：
Mathieu Chalvidal;Matthew Ricci;R. VanRullen;Thomas Serre
中科院分区：
文献类型：
--
作者：
Mathieu Chalvidal;Matthew Ricci;R. VanRullen;Thomas Serre
尽管它们有优雅的公式和轻量级的内存成本，神经常微分方程（node）仍然受到已知的表征限制。特别是，由节点学习的单个流不能表达从给定数据空间到自身的所有同态，并且与具有层相关权重的离散体系结构相比，它们的静态权重参数化限制了它们可以学习的函数类型。在这里，我们描述了一个名为神经控制ODE （N-CODE）的新模块，旨在提高节点的表达能力。N-CODE模块的参数是由初始或当前激活状态的可训练映射控制的动态变量，分别产生开环和闭环控制形式。单个模块足以学习自适应驱动神经表示的非自治流上的分布。我们提供了理论和经验证据，证明N-CODE绕过了以前模型的局限性，并展示了如何在几个领域中增加模型表现力。在监督学习中，我们证明了我们的框架在训练速度和测试准确性方面都比node取得了更好的性能。在无监督学习中，我们将这种控制视角应用于具有潜在转换流的图像自编码器，大大提高了普通模型的表示能力，并在CIFAR-10上实现了最先进的图像重建。
Despite their elegant formulation and lightweight memory cost, neural ordinary differential equations (NODEs) suffer from known representational limitations. In particular, the single flow learned by NODEs cannot express all homeomorphisms from a given data space to itself, and their static weight parametrization restricts the type of functions they can learn compared to discrete architectures with layer-dependent weights. Here, we describe a new module called neurally-controlled ODE (N-CODE) designed to improve the expressivity of NODEs. The parameters of N-CODE modules are dynamic variables governed by a trainable map from initial or current activation state, resulting in forms of open-loop and closed-loop control, respectively. A single module is sufficient for learning a distribution on non-autonomous flows that adaptively drive neural representations. We provide theoretical and empirical evidence that N-CODE circumvents limitations of previous models and show how increased model expressivity manifests in several domains. In supervised learning, we demonstrate that our framework achieves better performance than NODEs as measured by both training speed and testing accuracy. In unsupervised learning, we apply this control perspective to an image Autoencoder endowed with a latent transformation flow, greatly improving representational power over a vanilla model and leading to state-of-the-art image reconstruction on CIFAR-10.