Universal Estimation of Directed Information
Universal Estimation of Directed Information
复制标题
DOI：
10.1109/tit.2013.2267934
复制
发表时间：
2013-10-01
影响因子：
2.5
通讯作者：
Weissman, Tsachy
中科院分区：
文献类型：
--
作者：
Jiao, Jiantao;Permuter, Haim H.;Weissman, Tsachy
基于泛概率分配，给出了联合平稳遍历有限字母表过程之间有向信息率的四个估计式。第一种是Shannon-McMillan-Breiman类型的估计量，类似于Verdu和Cai等人在2005年使用的估计量。2006年用于估计其他信息衡量标准。我们证明了对于任何基本的普适概率分配，估计量的几乎必然和L-1收敛性质。其他三个估计器将普适概率分配映射到不同的泛函，每个泛函都表现出相对的优点，如光滑性、非负性和有界性。我们证明了这些估计在几乎必然意义下和L-1意义下的相合性，并在较温和的条件下得到了极小极大意义下的次最优收敛速度。这些估计量直接用于估计平稳遍历有限字母表过程的其他信息度量，如熵率和互信息率，具有接近最优的性能，并提供了现有文献中经典方法的替代方法。在这些理论结果的指导下，使用上下文树加权算法作为通用概率分配来实现所提出的估计器。通过对合成数据和真实数据的实验，验证了所提方案在实际应用中的潜力，以及有向信息估计在检测和测量因果影响和延迟方面的有效性。
Four estimators of the directed information rate between a pair of jointly stationary ergodic finite-alphabet processes are proposed, based on universal probability assignments. The first one is a Shannon-McMillan-Breiman-type estimator, similar to those used by Verdu in 2005 and Cai et al. in 2006 for estimation of other information measures. We show the almost sure and L-1 convergence properties of the estimator for any underlying universal probability assignment. The other three estimators map universal probability assignments to different functionals, each exhibiting relative merits such as smoothness, nonnegativity, and boundedness. We establish the consistency of these estimators in almost sure and L-1 senses, and derive near-optimal rates of convergence in the minimax sense under mild conditions. These estimators carry over directly to estimating other information measures of stationary ergodic finite-alphabet processes, such as entropy rate and mutual information rate, with near-optimal performance and provide alternatives to classical approaches in the existing literature. Guided by these theoretical results, the proposed estimators are implemented using the context-tree weighting algorithm as the universal probability assignment. Experiments on synthetic and real data are presented, demonstrating the potential of the proposed schemes in practice and the utility of directed information estimation in detecting and measuring causal influence and delay.