Statistical theory for Gaussian process function approximation based on theory of image restoration and its application

基于图像复原理论的高斯过程函数逼近统计理论及其应用

• 批准号: 14580438
• 负责人: OKADA Masato
• 金额: $ 2.37万
• 依托单位: RIKEN (The Institute of Physical and Chemical Research)
• 依托单位国家: 日本
• 项目类别: Grant-in-Aid for Scientific Research (C)
• 财政年份: 2002
• 资助国家: 日本
• 起止时间: 2002 至 2003
• 项目状态: 已结题
• 来源: https://kaken.nii.ac.jp/en/grant/KAKENHI-PROJECT-14580438/
• 关键词: function approximation; image restoration; Gaussian model; Fourier transformation; singularity; statistical mechanics; learning machine; on-line learning; 並進対称性

It had been usually assumed that the additive noise does not correlated with each other in the frameworks of the image restoration and function approximation. We instigated the use of the Baysian inference to restore noise-degraded images under conditions of spatially correlated noise. We obtained the expected value of a restored image and obtained the optimal hyper-parameters. We discussed whether the conventional spatially uncorrelated noise model could cope with the spatially correlated noise or not, and found it could not. Furthermore, we discussed the hyper-parameter estimation based on the maximum marginalized likelihood method, and found an iterative algorithm for obtaining the maximum can not be converged. We thought the reason is singularity in the model. Thus, we concentrated the parameter estimation for hierarchical models with singular structure. Using two-layer neural networks, we investigate influences of singularities on dynamics of standard gradient learning and natural gradient learning under various learning conditions. In the standard gradient learning, we found a quasi-plateau phenomenon, which is severer than the well known plateau in some cases. The slow convergence clue to the quasi-plateau and plateau becomes extremely serious when an optimal point is in a neighborhood of a singularity. In the natural gradient learning, however, the quasi-plateau and plateau are not observed and convergence speed is hardly affected by singularity.

在图像恢复和函数逼近的框架中，通常假设加性噪声彼此不相关。我们提倡使用贝叶斯推理在空间相关噪声的条件下恢复噪声退化的图像。我们获得了恢复图像的期望值并获得了最优的超参数。我们讨论了传统的空间不相关噪声模型是否可以处理空间相关噪声，结果发现它不能。此外，我们讨论了基于最大边缘化似然法的超参数估计，发现获取最大值的迭代算法无法收敛。我们认为原因是模型中的奇点。因此，我们集中了具有奇异结构的分层模型的参数估计。使用两层神经网络，我们研究了各种学习条件下奇点对标准梯度学习和自然梯度学习动态的影响。在标准梯度学习中，我们发现了一种准高原现象，在某些情况下比众所周知的高原现象更严重。当最优点位于奇点附近时，准平台和平台的缓慢收敛线索变得极其严重。然而，在自然梯度学习中，没有观察到准平台和平台，并且收敛速度几乎不受奇异性的影响。

项目成果

M.Inoue, H.Park, M.Okada: "On-line learning theory of soft committee machines with correlated hidden units. -Steepest gradient descent and natural gradient descent-"Journal of Physical Society of Japan. (印刷中).

M.Inoue、H.Park、M.Okada：“具有相关隐藏单元的软委员会机器的在线学习理论。-最速梯度下降和自然梯度下降-”日本物理学会杂志（正在出版）。

Tatsuto Murayama: "One-step RSB scheme for the rate distortion functions."Journal of Physics A: Mathematical and General. vol.36. 11123-11130 (2003)

Tatsuto Murayama：“速率畸变函数的一步 RSB 方案。”物理学杂志 A：数学与综合。

Kazuyuki Hara: "On-line learning through simple perceptron learning with a margin"Neural Networks,. vol.17, No.2. 215-223 (2004)

Kazuyuki Hara：“通过简单感知器学习进行在线学习”神经网络，。

Kazuyuki Hara, Masato Okada: "On-line learning through simple perceptron learning with a margin"Neural Networks. vol.17, No.2. 215-223 (2004)

Kazuyuki Hara、Masato Okada：“通过简单感知器学习进行在线学习”神经网络。

Hyeyoung Park, Masato Inoue, Masato Okada: "On-line learning dynamics of two-layer neural networks with unidentifiable parameters."Journal of Physics A : Mathematical and General. vol.36. 11753-11764 (2003)

Hyeyoung Park、Masato Inoue、Masato Okada：“具有不可识别参数的两层神经网络的在线学习动态。”《物理学杂志 A：数学与综合》。