Assisting Human Decisions in Document Matching
Assisting Human Decisions in Document Matching
复制标题
DOI：
10.48550/arxiv.2302.08450
复制
发表时间：
2023-02
期刊：
影响因子：
--
通讯作者：
Joon Sik Kim;Valerie Chen;Danish Pruthi;Nihar B. Shah;Ameet Talwalkar
中科院分区：
文献类型：
--
作者：
Joon Sik Kim;Valerie Chen;Danish Pruthi;Nihar B. Shah;Ameet Talwalkar
许多实际应用，从同行评审中的论文评审员分配到招聘中的求职者匹配，都需要人类决策者通过将他们的专业知识与机器学习模型的预测相结合来识别相关的匹配。在许多这样的模型辅助文档匹配任务中，决策者强调需要关于模型输出（或数据）的辅助信息来促进他们的决策。在本文中，我们设计了一个代理匹配任务，使我们能够评估哪种辅助信息提高决策者的性能（在准确性和时间方面）。通过一项众包（N=271名参与者）研究，我们发现，提供黑盒模型解释会降低用户在匹配任务上的准确性，这与人们普遍认为的通过更好地理解模型来帮助用户的观点相反。另一方面，定制的方法，旨在密切关注一些特定任务的desiderata被发现是有效的，在提高用户的性能。令人惊讶的是，我们还发现，用户的辅助信息的感知效用是不一致的，他们的客观效用（通过他们的任务性能测量）。
Many practical applications, ranging from paper-reviewer assignment in peer review to job-applicant matching for hiring, require human decision makers to identify relevant matches by combining their expertise with predictions from machine learning models. In many such model-assisted document matching tasks, the decision makers have stressed the need for assistive information about the model outputs (or the data) to facilitate their decisions. In this paper, we devise a proxy matching task that allows us to evaluate which kinds of assistive information improve decision makers' performance (in terms of accuracy and time). Through a crowdsourced (N=271 participants) study, we find that providing black-box model explanations reduces users' accuracy on the matching task, contrary to the commonly-held belief that they can be helpful by allowing better understanding of the model. On the other hand, custom methods that are designed to closely attend to some task-specific desiderata are found to be effective in improving user performance. Surprisingly, we also find that the users' perceived utility of assistive information is misaligned with their objective utility (measured through their task performance).