Singularities affect dynamics of learning in neuromanifolds
Singularities affect dynamics of learning in neuromanifolds
复制标题
DOI：
10.1162/neco.2006.18.5.1007
复制
发表时间：
2006-05-01
影响因子：
2.9
通讯作者：
Ozeki, Tomoko
中科院分区：
文献类型：
--
作者：
Amari, Shun-ichi;Park, Hyeyoung;Ozeki, Tomoko
分层系统的参数空间，如多层感知器，由于隐含单元的对称性和退化而包含奇点。参数空间形成几何流形，在神经网络的情况下称为神经流形。这样的模型用统计模型来辨识，黎曼度量由Fisher信息矩阵给出。然而，矩阵在奇点时退化。这种奇异结构不仅在多层感知器中普遍存在，而且在高斯混合概率密度、ARMA时间序列模型等许多情况下也普遍存在。Cramer-Rao定理的标准统计范式并不成立，奇异性导致了参数估计、假设检验、贝叶斯推理、模型选择，特别是从例子中学习的动力学方面的奇怪行为。到目前为止，主流的理论还没有对奇异性引起的问题给予太多的关注，只依赖于为正则(非奇异)模型发展的普通统计理论。本文综述了与多层感知器和高斯混合有关的统计流形的奇异性引起的现象。我们展示了我们在这些问题上的最新结果。简单的玩具模型也被用来显示明确的解决方案。我们解释说，最大似然估计不再服从高斯分布，甚至渐近服从高斯分布，因为Fisher信息矩阵退化，AIC，BIC和MDL等模型选择标准在这些模型中不成立，光滑的贝叶斯先验在这些模型中变得奇异，学习动力学的轨迹受到奇异性的强烈影响，导致参数空间中的平坦或缓慢的流形。由于考虑了奇异几何结构，自然梯度法具有较好的计算性能。通过算例对泛化误差和训练误差进行了研究。
The parameter spaces of hierarchical systems such as multilayer perceptrons include singularities due to the symmetry and degeneration of hidden units. A parameter space forms a geometrical manifold, called the neuromanifold in the case of neural networks. Such a model is identified with a statistical model, and a Riemannian metric is given by the Fisher information matrix. However, the matrix degenerates at singularities. Such a singular structure is ubiquitous not only in multilayer perceptrons but also in the gaussian mixture probability densities, ARMA time-series model, and many other cases. The standard statistical paradigm of the Cramer-Rao theorem does not hold, and the singularity gives rise to strange behaviors in parameter estimation, hypothesis testing, Bayesian inference, model selection, and in particular, the dynamics of learning from examples. Prevailing theories so far have not paid much attention to the problem caused by singularity, relying only on ordinary statistical theories developed for regular (nonsingular) models. Only recently have researchers remarked on the effects of singularity, and theories are now being developed.This article gives an overview of the phenomena caused by the singularities of statistical manifolds related to multilayer perceptrons and gaussian mixtures. We demonstrate our recent results on these problems. Simple toy models are also used to show explicit solutions. We explain that the maximum likelihood estimator is no longer subject to the gaussian distribution even asymptotically, because the Fisher information matrix degenerates, that the model selection criteria such as AIC, BIC, and MDL fail to hold in these models, that a smooth Bayesian prior becomes singular in such models, and that the trajectories of dynamics of learning are strongly affected by the singularity, causing plateaus or slow manifolds in the parameter space. The natural gradient method is shown to perform well because it takes the singular geometrical structure into account. The generalization error and the training error are studied in some examples.