Measuring Generalization with Optimal Transport
Measuring Generalization with Optimal Transport
复制标题
DOI：
--
复制
发表时间：
2021-06
期刊：
影响因子：
--
通讯作者：
Ching-Yao Chuang;Youssef Mroueh;K. Greenewald;A. Torralba;S. Jegelka
中科院分区：
文献类型：
--
作者：
Ching-Yao Chuang;Youssef Mroueh;K. Greenewald;A. Torralba;S. Jegelka
理解深度神经网络的泛化是深度学习中最重要的任务之一。虽然已经取得了很大的进展，但理论误差界限仍然经常表现得与经验观察不同。在这项工作中，我们开发了基于边界的泛化边界，其中边界用从训练分布中采样的独立随机子集之间的最优运输成本归一化。特别是，最优运输成本可以被解释为方差的泛化，它捕获了学习到的特征空间的结构属性。在给定训练数据和网络参数的情况下，我们的边界在大规模数据集上稳健地预测了泛化误差。从理论上讲，我们证明了特征的集中和分离在泛化中起着至关重要的作用，支持了文献中的实证结果。代码可在\url{https://github.com/chingyaoc/kV-Margin}上获得。
Understanding the generalization of deep neural networks is one of the most important tasks in deep learning. Although much progress has been made, theoretical error bounds still often behave disparately from empirical observations. In this work, we develop margin-based generalization bounds, where the margins are normalized with optimal transport costs between independent random subsets sampled from the training distribution. In particular, the optimal transport cost can be interpreted as a generalization of variance which captures the structural properties of the learned feature space. Our bounds robustly predict the generalization error, given training data and network parameters, on large scale datasets. Theoretically, we demonstrate that the concentration and separation of features play crucial roles in generalization, supporting empirical results in the literature. The code is available at \url{https://github.com/chingyaoc/kV-Margin}.