Minimizing α -Information for Generalization and Interpretation
Minimizing α -Information for Generalization and Interpretation
复制标题
最小化 α -用于概括和解释的信息
DOI：
10.1007/pl00013828
复制
发表时间：
1998
期刊：
影响因子：
1.1
通讯作者：
R. Kamimura
中科院分区：
文献类型：
--
作者：
R. Kamimura
本文提出了一种最小α信息量的方法来最大化和最小化隐藏单元中包含的信息。该方法旨在解释内部表示，提高泛化性能。α信息最小化强制隐藏单元具有最大信息或最小信息，这取决于隐藏单元的重要性。要解释内部表示，我们只需看到少量具有最大信息的隐藏单元，而忽略具有最小信息的隐藏单元。此外，通过最小化α信息，可以消除不必要的信息，从而导致更好的泛化。首先，将α信息应用于一个简单的数据压缩问题，在这个问题中，四个字符可以压缩成一个隐藏单元。然后，我们将α信息最小化应用于神经网络中最具挑战性的问题之一，即自然语言的描述和理解。作为通过神经网络获得描述充分性的一个例子，我们处理了一种接近英语的人工语言的良好性的推理。实验结果表明，该方法可以获得显式的内部表示，并能显著提高泛化能力。
In this paper we propose a minimumα-information method to maximize and minimize information contained in hidden units. The method aims to interpret internal representations and to improve generalization performance. Theα-information minimization forces hidden units to have maximum information or minimum information, depending on the importance of hidden units. To interpret internal representation, we have only to see a small number of hidden units with maximum information, ignoring hidden units with minimum information. In addition, by minimizing theα-information, unnecessary information can be eliminated, leading to better generalization. First,α-information was applied to a simple data compression problem in which four characters can be compressed into one hidden unit. Then we appliedα-information minimization to one of the most challenging topics in neural networks, namely, the description and understanding of natural languages. As an example to demonstrate the acquisition of descriptive adequacy by neural networks, we dealt with the inference of well-formedness of an artificial language close to English. Experimental results confirmed that explicit internal representations can be obtained and generalization can be significantly improved.