Realizing competitive learning by information maximization and its application to language acquisition

我们提出了一个新的信息理论的方法来神经计算。原则上，我们假设生命系统能够以一种策略性的方式控制信息，以保持它们在极端不确定的外部条件下的存在。例如，生命系统必须最大限度地增加外部环境的信息含量，以维持其生存，免受外部环境的破坏力。将这一原理应用于神经网络，我们提出了一种新的信息论竞争学习方法，它完全不同于传统的方法，竞争过程是通过最大化互信息来实现的。基于这种信息论竞争学习，我们开发了以下五种不同的神经计算方法。(1)成本敏感信息最大化：在最大化信息时，相关成本必须最小化。这种方法可以产生更忠实于输入模式的内部表示。(2)监督信息论竞争学习：将无监督信息论竞争学习扩展到监督学习。在这种方法中，我们使用多层网络，其中输出层被添加到第一竞争层。(3)教师指导学习：教师指导学习是一种新的监督学习方法，利用信息最大化，其中没有错误的反向传播。因此，该方法是非常有效的生物健全方法。在前面的方法中，我们使用了sigmoid激活函数，在增加信息方面存在一些问题。为了解决这个问题，我们使用高斯激活函数。(5)信息最大化和最小化的统一：我们试图统一信息最大化和最小化，因为这两种方法的缺点可以在这个框架中解决。在这个专题中，我们提供了战略信息控制的研究现状。

We propose a new information-theoretic approach to neural computing. In principle, we postulate that living systems manage to control information in a strategic way to keep their existence in extremely uncertain outer conditions. For example, living systems must maximize information content on outer environment to maintain their existence from destructive forces from outer environment. Applying this principle to neural networks, we have proposed a new information-theoretic competitive learning method that is completely different from conventional methods in that competition processes are realized by maximizing mutual information. Based upon this information-theoretic competitive learning, we have developed the following five different approaches to neural computing.(1) Cost-sensitive information maximization : in maximizing information, the associated cost must be minimized. This method can produce internal representations more faithful to input patterns.(2) Supervised information-theoretic competitive learning : unsupervised information-theoretic competitive learning is extended to supervised learning. In this method, we use multi-layered networks in which the output layer is added to the first competitive layer.(3) Teacher-directed learning: Teacher-directed learning is a new supervised learning method using information maximization in which no error back-propagation is used. Thus, the method is very efficient and biological sound method. In the previous method, we used the sigmoid activation functions with some problems in increasing information. To remedy this, we use the Gaiussian activation functions.(5) Unification of information maximization and minimization : We try to unify information maximization and minimization, because the shortcomings of both methods can be resolved in this framework. In this special session, we offer the present state of research of strategic information control.