Demystifying Disagreement-on-the-Line in High Dimensions
Demystifying Disagreement-on-the-Line in High Dimensions
复制标题
DOI：
10.48550/arxiv.2301.13371
复制
发表时间：
2023-01
期刊：
影响因子：
--
通讯作者：
Dong-Hwan Lee;Behrad Moniri;Xinmeng Huang;Edgar Dobriban;Hamed Hassani
中科院分区：
文献类型：
--
作者：
Dong-Hwan Lee;Behrad Moniri;Xinmeng Huang;Edgar Dobriban;Hamed Hassani
在分布偏移下评估机器学习模型的性能是具有挑战性的，特别是当我们只有来自偏移（目标）域的未标记数据沿着来自原始（源）域的标记数据时。最近的研究表明，不一致的概念，即两个用不同随机性训练的模型在相同输入上的差异程度，是解决这个问题的关键。实验表明，不一致性和预测误差之间存在很强的联系，这已经被用来估计模型的性能。实验发现了线上不一致现象，即目标域下的分类误差通常是源域下的分类误差的线性函数;只要这个性质成立，源域和目标域下的不一致就遵循相同的线性关系。在这项工作中，我们开发了一个理论基础，用于分析高维随机特征回归中的分歧，并研究在什么条件下，在我们的设置中发生的分歧在线现象。在CIFAR-10-C、Tiny ImageNet-C和Camelyon 17上的实验与我们的理论一致，并支持理论发现的普遍性。
Evaluating the performance of machine learning models under distribution shift is challenging, especially when we only have unlabeled data from the shifted (target) domain, along with labeled data from the original (source) domain. Recent work suggests that the notion of disagreement, the degree to which two models trained with different randomness differ on the same input, is a key to tackle this problem. Experimentally, disagreement and prediction error have been shown to be strongly connected, which has been used to estimate model performance. Experiments have led to the discovery of the disagreement-on-the-line phenomenon, whereby the classification error under the target domain is often a linear function of the classification error under the source domain; and whenever this property holds, disagreement under the source and target domain follow the same linear relation. In this work, we develop a theoretical foundation for analyzing disagreement in high-dimensional random features regression; and study under what conditions the disagreement-on-the-line phenomenon occurs in our setting. Experiments on CIFAR-10-C, Tiny ImageNet-C, and Camelyon17 are consistent with our theory and support the universality of the theoretical findings.