Quantifying Synergistic Information Using Intermediate Stochastic Variables
Quantifying Synergistic Information Using Intermediate Stochastic Variables
复制标题
DOI：
10.3390/e19020085
复制
发表时间：
2017-02-01
期刊：
影响因子：
2.7
通讯作者：
Sloot, Peter M. A.
中科院分区：
文献类型：
--
作者：
Quax, Rick;Har-Shemesh, Omri;Sloot, Peter M. A.
量化随机变量之间的协同效应是信息论中一个重要的开放问题。当多个来源一起预测一个结果变量比单一来源预测的总和更好时，信息协同就会发生。这是生物学中的一种基本现象，例如在神经元网络和细胞调控过程中，不同的信息流整合在一起产生单一的响应，但在社会合作过程中以及在机器学习的统计推理任务中也是如此。在这里，我们从第一性原理出发，提出了协同信息和协同信息的度量。所提出的方法依赖于所谓的协同随机变量(SRV)，它被构造为关于单个源变量的零互信息，但关于整个源变量集的非零互信息。我们证明了我们的测度的几个基本和所需的性质，包括界和可加性性质。此外，我们还证明了我们的措施的几个重要结果，包括不同类型的协同信息可能在相同的变量集之间共存的事实。给出了一个数值实现，我们用它来证明协同作用与对噪声的恢复能力有关。我们的措施可能是在研究多元信息理论及其众多应用方面向前迈出的显著一步。
Quantifying synergy among stochastic variables is an important open problem in information theory. Information synergy occurs when multiple sources together predict an outcome variable better than the sum of single-source predictions. It is an essential phenomenon in biology such as in neuronal networks and cellular regulatory processes, where different information flows integrate to produce a single response, but also in social cooperation processes as well as in statistical inference tasks in machine learning. Here we propose a metric of synergistic entropy and synergistic information from first principles. The proposed measure relies on so-called synergistic random variables (SRVs) which are constructed to have zero mutual information about individual source variables but non-zero mutual information about the complete set of source variables. We prove several basic and desired properties of our measure, including bounds and additivity properties. In addition, we prove several important consequences of our measure, including the fact that different types of synergistic information may co-exist between the same sets of variables. A numerical implementation is provided, which we use to demonstrate that synergy is associated with resilience to noise. Our measure may be a marked step forward in the study of multivariate information theory and its numerous applications.