Rational neural networks
Rational neural networks
复制标题
DOI：
--
复制
发表时间：
2020-04
期刊：
影响因子：
--
通讯作者：
N. Boull'e;Y. Nakatsukasa;Alex Townsend
中科院分区：
文献类型：
--
作者：
N. Boull'e;Y. Nakatsukasa;Alex Townsend
我们考虑具有有理激活函数的神经网络。深度学习架构中非线性激活函数的选择至关重要，并且严重影响神经网络的性能。我们根据网络复杂度建立了最佳界限，并证明了有理神经网络比深度指数较小的ReLU网络更有效地逼近光滑函数。有理激活函数的灵活性和平滑性使其成为ReLU的有吸引力的替代方案，正如我们通过数值实验所证明的那样。
We consider neural networks with rational activation functions. The choice of the nonlinear activation function in deep learning architectures is crucial and heavily impacts the performance of a neural network. We establish optimal bounds in terms of network complexity and prove that rational neural networks approximate smooth functions more efficiently than ReLU networks with exponentially smaller depth. The flexibility and smoothness of rational activation functions make them an attractive alternative to ReLU, as we demonstrate with numerical experiments.