Individually Conditional Individual Mutual Information Bound on Generalization Error
Individually Conditional Individual Mutual Information Bound on Generalization Error
复制标题
DOI：
10.1109/isit45174.2021.9518016
复制
发表时间：
2020-12
期刊：
影响因子：
--
通讯作者：
Ruida Zhou;C. Tian;Tie Liu
中科院分区：
文献类型：
--
作者：
Ruida Zhou;C. Tian;Tie Liu
结合Bu等人的误差分解技术，提出了一种新的泛化误差信息论上界。以及Steinke和Zakynsinou的条件互信息(CMI)构造。在之前的工作中，Haghifam等人。结合上述两种技术，提出了一个不同的界，我们称之为条件个体互信息界。然而，在简单的高斯设置下，CMI和CIMI的界在顺序上都比Bu等人的差。这一观察结果促使我们提出了新的界，它通过减少条件互信息中的条件项来克服这个问题。在建立这个界限的过程中，建立了一个条件解耦引理，这也导致了这些信息论界限之间有意义的二分法和比较。
We propose a new information-theoretic bound on generalization error based on a combination of the error decomposition technique of Bu et al. and the conditional mutual information (CMI) construction of Steinke and Zakynthinou. In a previous work, Haghifam et al. proposed a different bound combining the two aforementioned techniques, which we refer to as the conditional individual mutual information (CIMI) bound. However, in a simple Gaussian setting, both the CMI and the CIMI bounds are order-wise worse than that by Bu et al.. This observation motivated us to propose the new bound, which overcomes this issue by reducing the conditioning terms in the conditional mutual information. In the process of establishing this bound, a conditional decoupling lemma is established, which also leads to a meaningful dichotomy and comparison among these information-theoretic bounds.