Aresearch of statistical properties of singular models such as a multi-layer perceptron

众所周知，多层感知器是一种奇异模型，其中Fisher信息矩阵在某些情况下可以是奇异的。奇异性来自参数基函数，基函数的形状随参数基函数的不同而变化。为了揭示奇异模型的统计特性，在本研究中，我们重点研究了变基函数。在基函数输出约束下，得到了过拟合度的一个上界。例如，该限制可以通过限制多层感知器中的输入权重空间来实现。通过应用这一结果，我们表明，对于使用一个高斯单位的回归，在训练中对于宽度参数经常获得很小的值。另一方面，我们推导了从有限个正交函数集中选择基函数的学习机的训练误差和泛化误差的期望。这阐明了具有变基函数的机器的AIC型模型选择标准需要目标函数的信息。我们通过应用收缩方法来解决这个问题。从变基函数的角度出发，提出了一种非参数回归的收缩方法。该方法以较少的计算时间产生了具有较小泛化误差的机器。本研究的结果有助于阐明具有变基函数的机器的统计特性，从而可以得到多层感知器这样的奇异模型。

The multi-layer perceptron is known to be a singular model in which the Fisher information matrix can be singular in some cases. The singularity comes from parametric basis functions by which the shapes of basis functions vary. To reveal statistical properties of a singular model, in this research, we focus on the variable basis functions. We derived a upper bound of the degree of over-fitting under a restriction on basis function outputs. For example, the restriction can be achieved by restricting an input weight space in a multi-layer perceptron. By applying this result, we showed that, for a regression using one Gaussian unit, a very small value is frequently obtained for a width parameter in training. On the other hand, we derived the expectations of the training error and generalization error of learning machine in which basis functions which are chosen from a finite set of orthogonal functions. This clarifies that AIC type model selection criteria for machines with variable basis functions need an information of a target function. We solve this problem by applying a shrinkage method. From a viewpoint of variable basis functions, furthermore, we proposed a shrinkage method for a nonparametric regression. The method produces a machine with low generalization error in less computational time. The results obtained in this research helps for clarifying statistical properties of a machine with variable basis functions, thus, a singular model such as multi-layer perceptrons.