The Brier score does not evaluate the clinical utility of diagnostic tests or prediction models.
The Brier score does not evaluate the clinical utility of diagnostic tests or prediction models.
复制标题
DOI：
10.1186/s41512-017-0020-3
复制
发表时间：
2017-01-01
影响因子：
--
通讯作者：
Vickers, Andrew J
中科院分区：
文献类型：
--
作者：
Assel, Melissa;Sjoberg, Daniel D;Vickers, Andrew J
背景：各种统计数据已被提出作为工具，以帮助研究人员评估诊断测试或预测模型的价值。Brier评分被推荐的理由是，它是一个适当的评分规则，受区分和校准的影响。然而，Brier分数依赖于流行程度，因此测试或模型的排序可能不恰当地随流行程度而变化。方法：我们探讨了四种常见的临床情况：比较高度准确的二元试验与中等预测能力的连续预测模型；两种二元检测的比较，其中敏感性与特异性的重要性与患病率呈负相关；模型和测试与假设所有患者或没有患者呈阳性的默认策略的比较；并对两种不同方向标定误差的模型进行了比较。结果：在每种情况下，我们发现Brier评分给出了不适当的测试和模型排序。相反，净效益，一个决策分析的度量，给出的结果总是倾向于更优的测试或模型。结论：Brier评分不能评价诊断试验或预测模型的临床价值。作为另一种选择，我们主张使用决策分析措施，如净效益。试验注册：不适用。
BACKGROUND: A variety of statistics have been proposed as tools to help investigators assess the value of diagnostic tests or prediction models. The Brier score has been recommended on the grounds that it is a proper scoring rule that is affected by both discrimination and calibration. However, the Brier score is prevalence dependent in such a way that the rank ordering of tests or models may inappropriately vary by prevalence.METHODS: We explored four common clinical scenarios: comparison of a highly accurate binary test with a continuous prediction model of moderate predictiveness; comparison of two binary tests where the importance of sensitivity versus specificity is inversely associated with prevalence; comparison of models and tests to default strategies of assuming that all or no patients are positive; and comparison of two models with miscalibration in opposite directions.RESULTS: In each case, we found that the Brier score gave an inappropriate rank ordering of the tests and models. Conversely, net benefit, a decision-analytic measure, gave results that always favored the preferabletest or model.CONCLUSIONS: Brier score does not evaluate clinical value of diagnostic tests or prediction models. We advocate, as an alternative, the use of decision-analytic measures such as net benefit.TRIAL REGISTRATION: Not applicable.