Statistical criticality arises in most informative representations
Statistical criticality arises in most informative representations
复制标题
DOI：
10.1088/1742-5468/ab16c8
复制
发表时间：
2019-06-01
影响因子：
2.4
通讯作者：
Song, Juyong
中科院分区：
文献类型：
--
作者：
Cubero, Ryan John;Jo, Junghyo;Song, Juyong
我们表明，统计临界性，即发生的幂规律频率分布，出现在样本中，最大限度地提供了关于潜在的生成过程的信息。为了得出这一结论，我们首先确定了di的频率。相反的结果出现在样本中，因为变量携带着关于生成过程的有用信息。频率的熵，我们称之为相关性，为信息位的数量提供了一个上限。这不同于我们用来衡量分辨率的数据的熵。在给定分辨率下最大化相关性的样本--我们称之为信息量最大的样本--表现出统计上的关键。特别是，Zipf定律出现在分辨率(即压缩)和相关性之间的最佳权衡。作为副产品，我们导出了在缺乏关于生成模型的先验知识的情况下，可以从数据集中估计的最大参数个数的界。此外，我们还将临界性与数据生成过程表示的统计性质联系起来。我们证明，由于渐近均分性质的集中性，关于数据生成过程的最大信息量的表示被表征为能级的指数分布。这源于统计力学中的最小熵原理，即最大熵原理的共轭。这解释了为什么统计临界性不需要在信息量最大的样本中进行参数微调。
We show that statistical criticality, i.e. the occurrence of power law frequency distributions, arises in samples that are maximally informative about the underlying generating process. In order to reach this conclusion, we first identify the frequency with which di. erent outcomes occur in a sample, as the variable carrying useful information on the generative process. The entropy of the frequency, that we call relevance, provides an upper bound to the number of informative bits. This differs from the entropy of the data, that we take as a measure of resolution. Samples that maximise relevance at a given resolution-that we call maximally informative samples-exhibit statistical criticality. In particular, Zipf's law arises at the optimal trade-off between resolution (i.e. compression) and relevance. As a byproduct, we derive a bound of the maximal number of parameters that can be estimated from a dataset, in the absence of prior knowledge on the generative model.Furthermore, we relate criticality to the statistical properties of the representation of the data generating process. We show that, as a consequence of the concentration property of the asymptotic equipartition property, representations that are maximally informative about the data generating process are characterised by an exponential distribution of energy levels. This arises from a principle of minimal entropy, that is conjugate of the maximum entropy principle in statistical mechanics. This explains why statistical criticality requires no parameter fine tuning in maximally informative samples.