Evaluating the Impacts of Machine Learning Algorithms on Human Decisions

评估机器学习算法对人类决策的影响

• 批准号: 2051196
• 负责人: Kosuke Imai
• 金额: $ 33万
• 依托单位: Harvard University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-07-01 至 2024-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2051196&HistoricalAwards=false
• 关键词: Evaluating; Impacts; Machine; Learning; Algorithms

This research project will develop a methodological framework and set of tools for experimentally evaluating the impacts of machine learning algorithms on human decisions. In today's data-driven society, decisions often are based at least in part on algorithmic recommendations. Whenever choosing movies to watch or shopping for clothes to wear, online sites are constantly feeding consumers with such information. The project will develop methodologies to evaluate whether algorithmic recommendations help human decision makers achieve their goals and how they affect the fairness of such decisions. The new methodologies will help researchers empirically evaluate the efficacy of algorithm-assisted human decision making in a wide range of settings. These settings include individual decisions such as online shopping as well as decisions in medicine, finance, and judicial systems that have the potential to affect the lives of many in society. The investigators will apply the new methods to a randomized evaluation of pretrial risk assessment instruments on judicial decisions. An open-source software package will be developed, and the databases used in this research will be made publicly available.This project will develop tools for experimentally evaluating whether algorithmic recommendations help human decision makers achieve their goals and how such recommendations affect the fairness of such decisions. On the methodological front, the project will show how to evaluate the impacts of machine learning algorithms on the accuracy and fairness of human decisions. Although there exists a growing literature on algorithmic fairness, existing research almost exclusively focuses on the evaluation of accuracy and fairness of the algorithms themselves. Machines and humans have their own biases, however, and these biases may interact in unexpected ways to influence ultimate decisions. Also, the existing definitions of fairness do not account for the fact that decisions may influence individuals. The methodological framework to be developed will address these open problems. On the substantive front, the project will analyze data on original, real-world randomized controlled trials (RCTs) in collaboration with several jurisdictions in the United States. The project will analyze these RCTs to evaluate the impacts of pretrial risk assessment instruments (PRAIs) on judicial decisions. There has been a growing concern in the academic and public-policy communities about the potential racial bias of these PRAIs. This research will develop and implement rigorous evaluation methodologies to answer policy-relevant questions so that direct contributions can be made to this important public policy debate.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该研究项目将开发一个方法框架和一套工具，用于实验评估机器学习算法对人类决策的影响。在当今数据驱动的社会中，决策通常至少部分基于算法建议。每当选择看电影或购物的衣服穿，在线网站不断提供这些信息的消费者。该项目将开发方法来评估算法建议是否有助于人类决策者实现他们的目标，以及它们如何影响这些决策的公平性。新方法将帮助研究人员在广泛的环境中实证评估算法辅助人类决策的有效性。这些环境包括个人的决定，如网上购物以及医疗，金融和司法系统的决定，这些决定有可能影响社会中许多人的生活。研究人员将应用新方法对司法判决的审前风险评估工具进行随机评估。该项目将开发一个开源软件包，并将公开本研究中使用的数据库。该项目将开发工具，以实验方式评估算法建议是否有助于人类决策者实现其目标，以及这些建议如何影响此类决策的公平性。在方法论方面，该项目将展示如何评估机器学习算法对人类决策的准确性和公平性的影响。虽然有越来越多的文献算法的公平性，现有的研究几乎完全集中在算法本身的准确性和公平性的评价。然而，机器和人类都有自己的偏见，这些偏见可能会以意想不到的方式相互作用，影响最终的决策。此外，现有的公平定义没有考虑到决策可能影响个人的事实。有待制定的方法框架将解决这些未决问题。在实质性方面，该项目将与美国的几个司法管辖区合作，分析原始的真实世界随机对照试验（RCT）的数据。该项目将分析这些随机对照试验，以评估审前风险评估工具（PRAIs）对司法判决的影响。学术界和公共政策界越来越担心这些PRAIs的潜在种族偏见。这项研究将开发和实施严格的评估方法，以回答政策相关的问题，使直接贡献，可以作出这一重要的公共政策辩论。这一奖项反映了NSF的法定使命，并已被认为是值得通过评估使用基金会的知识价值和更广泛的影响审查标准的支持。