Locally Valid and Discriminative Prediction Intervals for Deep Learning Models
Locally Valid and Discriminative Prediction Intervals for Deep Learning Models
复制标题
DOI：
--
复制
发表时间：
2021-06
期刊：
影响因子：
--
通讯作者：
Zhen Lin;Shubhendu Trivedi;Jimeng Sun
中科院分区：
文献类型：
--
作者：
Zhen Lin;Shubhendu Trivedi;Jimeng Sun
在深度学习模型中为关键的现实世界应用建立信任的关键是有效且理论上合理的不确定性量化，这是一项仍然具有挑战性的任务。有用的不确定性信息预计有两个关键属性：它应该是有效的（保证覆盖率）和判别性（当预期风险高时更不确定）。此外，当与深度学习（DL）方法结合时，它应该是可扩展的，并且对DL模型性能的影响最小。大多数现有的贝叶斯方法缺乏频率覆盖保证，通常会影响模型的性能。少数可用的频率论方法很少有歧视性和/或违反覆盖保证，由于不切实际的假设。此外，许多方法是昂贵的，或者需要对基本神经网络进行大量修改。基于保形预测的最新进展[13，33]并利用核回归的经典思想，我们提出了局部有效和判别预测区间（LVD），这是一种简单，高效和轻量级的方法，可以为几乎任何DL模型构建判别预测区间（PI）。由于没有对数据分布的假设，这样的PI还提供有限样本的局部覆盖保证（与更简单的边际覆盖相比）。我们经验验证，使用不同的数据集，除了是唯一的本地有效的方法DL，LVD也超过或匹配现有的不确定性量化方法的性能（包括覆盖率和预测精度），同时提供额外的好处，在可扩展性和灵活性。
Crucial for building trust in deep learning models for critical real-world applications is efficient and theoretically sound uncertainty quantification, a task that continues to be challenging. Useful uncertainty information is expected to have two key properties: It should be valid (guaranteeing coverage) and discriminative (more uncertain when the expected risk is high). Moreover, when combined with deep learning (DL) methods, it should be scalable and affect the DL model performance minimally. Most existing Bayesian methods lack frequentist coverage guarantees and usually affect model performance. The few available frequentist methods are rarely discriminative and/or violate coverage guarantees due to unrealistic assumptions. Moreover, many methods are expensive or require substantial modifications to the base neural network. Building upon recent advances in conformal prediction [13, 33] and leveraging the classical idea of kernel regression, we propose Locally Valid and Discriminative prediction intervals (LVD), a simple, efficient and lightweight method to construct discriminative prediction intervals (PIs) for almost any DL model. With no assumptions on the data distribution, such PIs also offer finite-sample local coverage guarantees (contrasted to the simpler marginal coverage). We empirically verify, using diverse datasets, that besides being the only locally valid method for DL, LVD also exceeds or matches the performance (including coverage rate and prediction accuracy) of existing uncertainty quantification methods, while offering additional benefits in scalability and flexibility.