NONLINEAR NEURONS IN THE LOW-NOISE LIMIT - A FACTORIAL CODE MAXIMIZES INFORMATION-TRANSFER
NONLINEAR NEURONS IN THE LOW-NOISE LIMIT - A FACTORIAL CODE MAXIMIZES INFORMATION-TRANSFER
复制标题
DOI：
10.1088/0954-898x/5/4/008
复制
发表时间：
1994-11-01
影响因子：
7.8
通讯作者：
PARGA, N
中科院分区：
文献类型：
--
作者：
NADAL, JP;PARGA, N
我们研究了在简单神经网络（一个输入层，一个输出层）中最大化信息传递的后果，重点关注非线性传递函数的情况。我们假设感受野（突触功效）和传递函数都可以适应环境。主要结果是，对于有界和可逆传递函数，在加性输出噪声消失且没有输入噪声的情况下，信息最大化（Linsker 的 infomax 原理）会导致阶乘代码 - 因此得到与 Barlow 冗余减少原理所需的相同解决方案。我们还表明，这一结果对于线性传递函数（更一般而言，无界传递函数）是有效的，前提是在加性约束下执行优化，即可以将其写为项之和，其中每一项都特定于一个输出神经元。最后，我们研究非零输入噪声的影响。我们发现，对于输入噪声的一阶，假设与（小）输出噪声相比较小，只要输出噪声从一个神经元到另一个神经元不相关，上述结果仍然有效。
We investigate the consequences of maximizing information transfer in a simple neural network (one input layer, one output layer), focusing on the case of nonlinear transfer functions. We assume that both receptive fields (synaptic efficacies) and transfer functions can be adapted to the environment. The main result is that, for bounded and invertible transfer functions, in the case of a vanishing additive output noise, and no input noise, maximization of information (Linsker's infomax principle) leads to a factorial code - hence to the same solution as required by the redundancy-reduction principle of Barlow. We also show that this result is valid for linear and, more generally, unbounded, transfer functions, provided optimization is performed under an additive constraint, i.e. which can be written as a sum of terms, each one being specific to one output neuron. Finally, we study the effect of a non-zero input noise. We find that, to first order in the input noise, assumed to be small in comparison with the (small) output noise, the above results are still valid, provided the output noise is uncorrelated from one neuron to the other.