Mathematical foundation of Singular Learning Machines based on Algebraic Geometry and Analysis

基于代数几何与分析的奇异学习机数学基础

• 批准号: 12680370
• 负责人: WATANABE Sumio
• 金额: $ 2.5万
• 依托单位: Tokyo Institute of Technology
• 依托单位国家: 日本
• 项目类别: Grant-in-Aid for Scientific Research (C)
• 财政年份: 2000
• 资助国家: 日本
• 起止时间: 2000 至 2002
• 项目状态: 已结题
• 来源: https://kaken.nii.ac.jp/grant/KAKENHI-PROJECT-12680370/
• 关键词: Singularities; algebraic geometry; algebraic analysis; Learning Theory; generalization error; stochastic complexity; 神経回路網; ニューラルネットワーク; 佐藤b関数; 自由エネルギー; 予測誤差; 特異点解消

A lot of learning machines are nonidentifiable, for example, artificial neural networks, normal mixtures, Bayesian networks, reduced rank approximations. In such learning machines, the mapping from the parameter to the probability distribution is not one-to-one, resulting that the Fisher information matrices are singular. It should be emphasized that there is no learning theory which can be applied to such singular learning machines. In this research, we developed a new learning theory which enables us to clarify the generalization errors of such learning machines. The main result is as follows. The generalization error that is defined as the average Kullback information from the true distribution to the estimated distribution can be asymptotically expanded as "a log n - (b-1)loglog n + c", where a, b, and c are constants. The important constants a and b are determined as the largest pole and its order of the zeta function of the Kullback in formation and the prior. They can be calculated by the blowing-up technology, which is well known in resolution of singularities in algebraic geometry.

许多学习机器是不可识别的，例如，人工神经网络，正常混合，贝叶斯网络，降秩近似。在这种学习机中，从参数到概率分布的映射不是一对一的，导致Fisher信息矩阵是奇异的。需要强调的是，目前还没有一种学习理论可以适用于这种奇异的学习机器。在这项研究中，我们开发了一种新的学习理论，使我们能够澄清这种学习机的泛化误差。主要结果如下。定义为真实分布到估计分布的平均Kullback信息的泛化误差可以渐近展开为“a log n - (b-1)loglog n + c”，其中a、b、c为常数。将重要常数a和b确定为Kullback信息和先验zeta函数的最大极点及其阶数。它们可以通过爆破技术来计算，爆破技术在代数几何奇点的求解中是众所周知的。

项目成果

K.Yamazaki, S.Watanabe: "A probabilistic algorithm to calculate the learning curves of hierarchical learning machines with singularities"Trans.on IEICE. J85-D-II, No.3. 363-372 (2002)

K.Yamazaki、S.Watanabe：“计算具有奇点的分层学习机的学习曲线的概率算法”Trans.on IEICE。

K. Watanabe, S. Watanabe: "On the Bayes generalization error of the reduced rank regression"IEICE Trans.. To appear, J86-A, No.3. (2003)

K. Watanabe，S. Watanabe：“关于降阶回归的贝叶斯泛化误差”IEICE Trans.. 出现，J86-A，No.3。

Sumio Watanabe, Shun-ichi Amari: "Learning coefficients of layered models when the true distribution mismatches the singularities"Neural Computation. 15・3(印刷中). (2003)

Sumio Watanabe、Shun-ichi Amari：“真实分布与奇点不匹配时的分层模型的学习系数”15・3（出版中）。

S. Watanabe: "Learning efficiency of redundant neural networks in Bayesian estimation"IEEE Transactions on Neural Networks. 12, No.6. 1475-1486 (2001)

S. Watanabe：“贝叶斯估计中冗余神经网络的学习效率”IEEE Transactions on Neural Networks。

渡邊 澄夫: "特異点を持つ学習モデルと事前分布の代数幾何"人工知能学会誌. 16巻2号. 308-315 (2001)

Sumio Watanabe：“具有奇点和先验分布的代数几何的学习模型”，日本人工智能学会杂志，第 16 卷，第 2. 308-315 期（2001 年）。