Leveraging Expert Consistency to Improve Algorithmic Decision Support
Leveraging Expert Consistency to Improve Algorithmic Decision Support
复制标题
利用专家一致性来改进算法决策支持
DOI：
10.48550/arxiv.2311.12582
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
A. Chouldechova
中科院分区：
文献类型：
--
作者：
Maria De;A. Dubrawski;A. Chouldechova
机器学习（ML）越来越多地用于支持高风险决策。然而，通常存在一个构造差距：决策任务感兴趣的构造与用作标签来训练ML模型的代理中捕获的内容之间的差距。因此，ML模型可能无法捕获决策标准的重要维度，从而阻碍了它们用于决策支持的实用性。因此，设计决策支持ML系统的一个重要步骤是在可用的代理中选择目标标签。在这项工作中，我们探讨了使用历史专家的决定作为一个丰富的-但也不完美-的信息来源，可以与观察到的结果相结合，以缩小结构差距。我们认为，管理者和系统设计者可能有兴趣从专家学习的情况下，他们表现出一致性，而从观察到的结果，否则。我们开发了一种方法，使这一目标使用的信息，通常是在组织信息系统。这涉及两个核心步骤。首先，我们提出了一种基于影响函数的方法来间接估计专家一致性时，数据中的每个情况下，由一个单一的专家进行评估。其次，我们引入了一种标签融合方法，允许ML模型同时从专家决策和观察结果中学习。我们的实证评估，使用模拟在临床环境和现实世界的数据，从儿童福利领域，表明所提出的方法成功地缩小了结构差距，产生更好的预测性能比学习无论是观察到的结果或专家的决定。
Machine learning (ML) is increasingly being used to support high-stakes decisions. However, there is frequently a construct gap: a gap between the construct of interest to the decision-making task and what is captured in proxies used as labels to train ML models. As a result, ML models may fail to capture important dimensions of decision criteria, hampering their utility for decision support. Thus, an essential step in the design of ML systems for decision support is selecting a target label among available proxies. In this work, we explore the use of historical expert decisions as a rich -- yet also imperfect -- source of information that can be combined with observed outcomes to narrow the construct gap. We argue that managers and system designers may be interested in learning from experts in instances where they exhibit consistency with each other, while learning from observed outcomes otherwise. We develop a methodology to enable this goal using information that is commonly available in organizational information systems. This involves two core steps. First, we propose an influence function-based methodology to estimate expert consistency indirectly when each case in the data is assessed by a single expert. Second, we introduce a label amalgamation approach that allows ML models to simultaneously learn from expert decisions and observed outcomes. Our empirical evaluation, using simulations in a clinical setting and real-world data from the child welfare domain, indicates that the proposed approach successfully narrows the construct gap, yielding better predictive performance than learning from either observed outcomes or expert decisions alone.
儿童福利工作者如何减少算法决策中的种族差异
DOI：
10.1145/3491102.3501831
发表时间：
2022
期刊：
Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems
影响因子：
--
作者：
Cheng, Hao-Fei;Stapleton, Logan;Kawakami, Anna;Sivaraman, Venkatesh;Cheng, Yanghuidi;Qing, Diana;Perer, Adam;Holstein, Kenneth;Wu, Zhiwei Steven;Zhu, Haiyi
通讯作者：
Zhu, Haiyi