JAWS: Auditing Predictive Uncertainty Under Covariate Shift
JAWS: Auditing Predictive Uncertainty Under Covariate Shift
复制标题
DOI：
--
复制
发表时间：
2022-07
期刊：
影响因子：
--
通讯作者：
Drew Prinster;Anqi Liu;S. Saria
中科院分区：
文献类型：
--
作者：
Drew Prinster;Anqi Liu;S. Saria
针对协变量平移下的无分布不确定性量化任务，提出了一系列包装方法-.JAWS还包括使用高阶影响函数的JAW的计算效率的\extbf{A}近似：\extbf{JAWA}。理论上，我们证明了JAW放宽了折刀+S关于数据可交换性的假设，即使在协变量平移的情况下也能实现相同的有限样本覆盖保证。在一般正则性假设下，Jawa进一步讨论了在样本容量或影响函数阶数限制下的JAW保证。此外，我们提出了一种通用的方法来改变预测区间生成方法及其对反向任务的保证：基于用户指定的误差标准，例如真实标签附近的安全或可接受的容差阈值，估计预测错误的概率。然后，我们提出了针对这个错误评估任务的重新定位的方法-.实际上，对于区间生成和误差评估预测性不确定性审计任务，JAWS在各种有偏见的真实世界数据集上的表现优于最先进的预测推理基线。
We propose \textbf{JAWS}, a series of wrapper methods for distribution-free uncertainty quantification tasks under covariate shift, centered on the core method \textbf{JAW}, the \textbf{JA}ckknife+ \textbf{W}eighted with data-dependent likelihood-ratio weights. JAWS also includes computationally efficient \textbf{A}pproximations of JAW using higher-order influence functions: \textbf{JAWA}. Theoretically, we show that JAW relaxes the jackknife+'s assumption of data exchangeability to achieve the same finite-sample coverage guarantee even under covariate shift. JAWA further approaches the JAW guarantee in the limit of the sample size or the influence function order under common regularity assumptions. Moreover, we propose a general approach to repurposing predictive interval-generating methods and their guarantees to the reverse task: estimating the probability that a prediction is erroneous, based on user-specified error criteria such as a safe or acceptable tolerance threshold around the true label. We then propose \textbf{JAW-E} and \textbf{JAWA-E} as the repurposed proposed methods for this \textbf{E}rror assessment task. Practically, JAWS outperform state-of-the-art predictive inference baselines in a variety of biased real world data sets for interval-generation and error-assessment predictive uncertainty auditing tasks.