On Leave-One-Out Conditional Mutual Information For Generalization
On Leave-One-Out Conditional Mutual Information For Generalization
复制标题
DOI：
10.48550/arxiv.2207.00581
复制
发表时间：
2022-07
期刊：
影响因子：
--
通讯作者：
M. Rammal;A. Achille;Aditya Golatkar;S. Diggavi;S. Soatto
中科院分区：
文献类型：
--
作者：
M. Rammal;A. Achille;Aditya Golatkar;S. Diggavi;S. Soatto
基于一种新的留一条件互信息（loove-one-out conditional mutual information，loo-CMI）度量，推导出监督学习算法的信息论泛化界。与其他CMI边界相反，这些边界是黑盒边界，不利用问题的结构，并且在实践中可能难以评估，我们的loo-CMI边界可以很容易地计算，并且可以结合其他概念进行解释，例如经典的留一交叉验证，优化算法的稳定性和损失景观的几何形状。它既适用于训练算法的输出，也适用于它们的预测。我们通过评估其在深度学习场景中的预测泛化差距来实证验证边界的质量。特别是，我们的界限是非空的大规模图像分类任务。
We derive information theoretic generalization bounds for supervised learning algorithms based on a new measure of leave-one-out conditional mutual information (loo-CMI). Contrary to other CMI bounds, which are black-box bounds that do not exploit the structure of the problem and may be hard to evaluate in practice, our loo-CMI bounds can be computed easily and can be interpreted in connection to other notions such as classical leave-one-out cross-validation, stability of the optimization algorithm, and the geometry of the loss-landscape. It applies both to the output of training algorithms as well as their predictions. We empirically validate the quality of the bound by evaluating its predicted generalization gap in scenarios for deep learning. In particular, our bounds are non-vacuous on large-scale image-classification tasks.