Improving generalization performance by information minimization
Improving generalization performance by information minimization
复制标题
通过信息最小化提高泛化性能
DOI：
10.1109/icnn.1994.374153
复制
发表时间：
1995
期刊：
影响因子：
--
通讯作者：
S. Nakanishi
中科院分区：
文献类型：
--
作者：
R. Kamimura;T. Takagi;S. Nakanishi
在本文中，我们试图证明，在网络能够产生适当精度的目标的条件下，存储在网络中的信息必须尽可能小，以提高泛化性能。信息由最大熵或不确定性与观测到的熵之差来定义。借鉴模糊熵的定义，为内部表示定义不确定性函数，表示为：-/spl upsi//sub i/ log /spl upsi//sub i/-(1-/spl upsi//sub i/) log (1-/spl upsi//sub i/)，其中/spl upsi//sub i/是一个隐藏单元活动。在制定了信息最小化的更新规则后，我们将该方法应用于语言习得问题：规则动词过去时形式的推断。实验结果证实，我们的方法显著减少了信息，大大提高了泛化性能。<<ETX>>
In this paper, we attempt to show that the information stored in networks must be as small as possible for the improvement of the generalization performance under the condition that the networks can produce targets with appropriate accuracy. The information is defined by the difference between maximum entropy or uncertainty and observed entropy. Borrowing a definition of fuzzy entropy, the uncertainty function is defined for the internal representation and represented by the equation: -/spl upsi//sub i/ log /spl upsi//sub i/-(1-/spl upsi//sub i/) log (1-/spl upsi//sub i/), where /spl upsi//sub i/ is a hidden unit activity. After having formulated an update rule for the minimization of the information, we applied the method to a problem of language acquisition: the inference of the past tense forms of regular verbs. Experimental results confirmed that by our method, the information was significantly decreased and the generalization performance was greatly improved.<<ETX>>