Selective Ensembles for Consistent Predictions
Selective Ensembles for Consistent Predictions
复制标题
DOI：
--
复制
发表时间：
2021-11
期刊：
影响因子：
--
通讯作者：
Emily Black;Klas Leino;Matt Fredrikson
中科院分区：
文献类型：
--
作者：
Emily Black;Klas Leino;Matt Fredrikson
最近的研究表明，为同一目标训练的模型，在一致的测试数据上实现了相似的准确性度量，但在个别预测上的表现可能非常不同。这种不一致性在高风险的情况下是不可取的，例如医疗诊断和金融。我们发现，这种不一致的行为超出了预测的功能属性，这同样可能有负面影响的模型的可理解性，以及一个人的能力，以找到追索权的科目。然后，我们引入选择性集合，通过对使用随机选择的起始条件训练的一组模型的预测应用假设检验来减轻这种不一致;重要的是，选择性集合可以在无法达到指定置信水平的一致结果的情况下弃权。我们证明了选择性合奏之间的预测分歧是有界的，并实证表明，选择性合奏实现一致的预测和功能属性，同时保持低的预测率。在几个基准数据集上，选择性集合达到零不一致的预测点，预测率低至1.5%。
Recent work has shown that models trained to the same objective, and which achieve similar measures of accuracy on consistent test data, may nonetheless behave very differently on individual predictions. This inconsistency is undesirable in high-stakes contexts, such as medical diagnosis and finance. We show that this inconsistent behavior extends beyond predictions to feature attributions, which may likewise have negative implications for the intelligibility of a model, and one's ability to find recourse for subjects. We then introduce selective ensembles to mitigate such inconsistencies by applying hypothesis testing to the predictions of a set of models trained using randomly-selected starting conditions; importantly, selective ensembles can abstain in cases where a consistent outcome cannot be achieved up to a specified confidence level. We prove that that prediction disagreement between selective ensembles is bounded, and empirically demonstrate that selective ensembles achieve consistent predictions and feature attributions while maintaining low abstention rates. On several benchmark datasets, selective ensembles reach zero inconsistently predicted points, with abstention rates as low 1.5%.