Neural Arithmetic Units
Neural Arithmetic Units
复制标题
神经算术单元
DOI：
--
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
alexander rosenberg johansen
中科院分区：
文献类型：
--
作者：
Andreas Madsen;alexander rosenberg johansen
神经网络可以近似复杂的函数，但它们很难对真实的数字执行精确的算术运算。算术运算缺乏归纳偏差，使得神经网络没有必要的底层逻辑来推断加法、减法和乘法等任务。我们提出了两个新的神经网络组件：神经加法单元（NAU），它可以学习精确的加法和减法;神经乘法单元（NMU），它可以乘以向量的子集。据我们所知，NMU是第一个可以学习在隐藏大小很大时从向量中相乘元素的算术神经网络组件。这两个新的组件的灵感来自最近提出的算术组件的理论分析。我们发现，仔细的初始化，限制参数空间，稀疏正则化是很重要的优化NAU和NMU时。与以前的神经单元相比，我们提出的单元NAU和NMU收敛更一致，参数更少，学习速度更快，可以收敛更大的隐藏大小，获得稀疏和有意义的权重，并且可以外推到负值和小值。
Neural networks can approximate complex functions, but they struggle to perform exact arithmetic operations over real numbers. The lack of inductive bias for arithmetic operations leaves neural networks without the underlying logic necessary to extrapolate on tasks such as addition, subtraction, and multiplication. We present two new neural network components: the Neural Addition Unit (NAU), which can learn exact addition and subtraction; and the Neural Multiplication Unit (NMU) that can multiply subsets of a vector. The NMU is, to our knowledge, the first arithmetic neural network component that can learn to multiply elements from a vector, when the hidden size is large. The two new components draw inspiration from a theoretical analysis of recently proposed arithmetic components. We find that careful initialization, restricting parameter space, and regularizing for sparsity is important when optimizing the NAU and NMU. Our proposed units NAU and NMU, compared with previous neural units, converge more consistently, have fewer parameters, learn faster, can converge for larger hidden sizes, obtain sparse and meaningful weights, and can extrapolate to negative and small values.