Cranky comments: detecting clinical decision support malfunctions through free-text override reasons
Cranky comments: detecting clinical decision support malfunctions through free-text override reasons
复制标题
DOI：
10.1093/jamia/ocy139
复制
发表时间：
2019-01-01
影响因子：
6.4
通讯作者：
Wright, Adam
中科院分区：
文献类型：
--
作者：
Aaron, Skye;McEvoy, Dustin S.;Wright, Adam
背景：基于规则的临床决策支持警报是已知的故障，但发现故障的工具是有限的。目的：探讨用户覆盖评论是否可以用于发现故障。方法：基于重写注释的样本，我们手动将数据库中至少有10条重写注释的所有规则分为3类：“破坏”，“未破坏，但可以改进”和“未破坏”。我们使用了3种方法（评论频率，古怪的词列表启发式，以及在评论样本上训练的朴素贝叶斯分类器）来根据其覆盖评论的特征自动对规则进行排名。我们使用人工分类作为真理来评估每个排名。结果：在所调查的规则中，有62条违规，13条有待改善，其余45条未违规。评论频率的表现不如随机排名，精度为20 / 8，AUC = 0.487。cranky评论启发式的表现更好，精度为20 / 16，AUC = 0.723。朴素贝叶斯分类器的精度为20 / 17，AUC = 0.738。讨论：覆盖评论发现了我们系统中26%的有效规则的故障。这是总故障的下限，远高于预期。即使对于资源不足的组织，查看由古怪的单词列表启发式识别的评论也可能是发现故障警报的有效和可行的方法。结论：覆盖注释是一个丰富的数据源，用于发现损坏或可以改进的警报。如果可能的话，我们建议定期监视所有重写注释。
Background: Rule-base clinical decision support alerts are known to malfunction, but tools for discovering malfunctions are limited. Objective: Investigate whether user override comments can be used to discover malfunctions.Methods: We manually classified all rules in our database with at least 10 override comments into 3 categories based on a sample of override comments: "broken," "not broken, but could be improved," and "not broken." We used 3 methods (frequency of comments, cranky word list heuristic, and a Naive Bayes classifier trained on a sample of comments) to automatically rank rules based on features of their override comments. We evaluated each ranking using the manual classification as truth.Results: Of the rules investigated, 62 were broken, 13 could be improved, and the remaining 45 were not broken. Frequency of comments performed worse than a random ranking, with precision at 20 of 8 and AUC = 0.487. The cranky comments heuristic performed better with precision at 20 of 16 and AUC = 0.723. The Naive Bayes classifier had precision at 20 of 17 and AUC = 0.738.Discussion: Override comments uncovered malfunctions in 26% of all rules active in our system. This is a lower bound on total malfunctions and much higher than expected. Even for low-resource organizations, reviewing comments identified by the cranky word list heuristic may be an effective and feasible way of finding broken alerts.Conclusion: Override comments are a rich data source for finding alerts that are broken or could be improved. If possible, we recommend monitoring all override comments on a regular basis.