EAGER: AI-DCL: Hybrid human algorithm predictions: balancing effort, accuracy, and perceived autonomy

EAGER：AI-DCL：混合人类算法预测：平衡努力、准确性和感知自主性

• 批准号: 1927245
• 负责人: Mark Steyvers
• 金额: $ 29.39万
• 依托单位: University of California-Irvine
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-08-15 至 2022-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1927245&HistoricalAwards=false
• 关键词: EAGER; AI; DCL; Hybrid; human

Artificial intelligence algorithms are becoming an increasingly common component in everyday decision-making scenarios. Such algorithms use medical data to assist doctors in making diagnostic and treatment decisions, use sensor data to assist drivers in operating vehicles, and use review data to make recommendations for products to purchase and restaurants to visit. These systems are often hybrid, with decisions influenced by a combination of an artificial intelligence algorithm and a human decision-maker. For example, in one mode of operation, a human makes each decision after taking into account input from the algorithm. In another mode of operation, when human resources are limited (as in medical diagnosis), a hybrid system can use the algorithm to make the majority of routine decisions and allocate to a human decision-maker only those problems that are most challenging to the algorithm. As these hybrid systems are being increasingly used in critical decision-making tasks it is important to address questions about the understanding and design of such systems. This project will address a series of fundamental questions about hybrid decision-making systems from the dual perspectives of algorithm development and cognitive models of human reasoning and decision-making. For example, how can the overall performance of a hybrid human-algorithm system be optimized when taking into account limited human resources and potential trade-offs between human and algorithmic skills? From a psychological perspective, does a perceived lack of autonomy negatively affect human engagement and can systems be designed to mitigate this? The project will develop new theories and methods for how humans and algorithms can work together and the results will help produce more accurate and more robust decision-making systems across a variety of areas such as medicine, transportation, business, and consumer applications. To achieve its goals the research project will bring together prior threads of work from psychology, machine learning, and Bayesian estimation. The project will consist of two closely-coupled components with a common focus on modeling and understanding of prediction problems that are handled by a combination of human and algorithmic expertise. The first component will develop and evaluate different computational and statistical frameworks for an algorithmic arbitrator that balances black-box predictions and human expertise in large-scale classification tasks. The second component will build on theories from human cognition and psychology to analyze joint algorithm-human prediction performance, with explicit consideration of the effect of a human dropping out and not continuing to work with the algorithm (e.g., due to a perceived lack of autonomy). An extensive series of user studies, under a variety of hybrid prediction scenarios and different decision allocation methods, will be conducted during the project to support the development of new cognitive insights and computational approaches for hybrid algorithm-human prediction systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

人工智能算法正在成为日常决策场景中越来越常见的组件。这些算法使用医疗数据来帮助医生做出诊断和治疗决策，使用传感器数据来帮助驾驶员操作车辆，并使用评论数据来推荐购买的产品和参观的餐馆。这些系统通常是混合的，决策受到人工智能算法和人类决策者的组合影响。例如，在一种操作模式中，人在考虑来自算法的输入之后做出每个决定。在另一种操作模式中，当人力资源有限时（如在医疗诊断中），混合系统可以使用算法来做出大多数例行决策，并仅将那些对算法最具挑战性的问题分配给人类决策者。随着这些混合系统越来越多地用于关键的决策任务，重要的是要解决有关的理解和设计这样的系统的问题。该项目将从算法开发和人类推理和决策的认知模型的双重角度解决有关混合决策系统的一系列基本问题。例如，当考虑到有限的人力资源和人类与算法技能之间的潜在权衡时，如何优化混合人类算法系统的整体性能？从心理学的角度来看，缺乏自主性是否会对人类参与产生负面影响，是否可以设计系统来减轻这种影响？该项目将为人类和算法如何协同工作开发新的理论和方法，其结果将有助于在医疗、交通、商业和消费者应用等各个领域产生更准确、更强大的决策系统。 为了实现其目标，该研究项目将汇集来自心理学，机器学习和贝叶斯估计的先前工作线程。该项目将由两个紧密耦合的组件组成，共同关注由人类和算法专业知识相结合处理的预测问题的建模和理解。第一部分将开发和评估不同的计算和统计框架，用于算法仲裁器，该仲裁器在大规模分类任务中平衡黑箱预测和人类专业知识。第二部分将建立在人类认知和心理学的理论基础上，以分析联合算法-人类预测性能，明确考虑人类退出并且不继续使用算法的影响（例如，由于缺乏自主权）。在项目期间，将在各种混合预测情景和不同决策分配方法下进行一系列广泛的用户研究，以支持开发新的认知见解和混合算法-人类预测系统的计算方法。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

项目成果

Can I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian Inference

我可以相信我的公平指标吗？

• 发表时间: 2020
• 期刊: Advances in Neural Information Processing Systems
• 作者: Smyth, P;Steyvers, M
• 通讯作者: Steyvers, M

A Brief Tour of Deep Learning from a Statistical Perspective

从统计角度简要介绍深度学习

• DOI: 10.1146/annurev-statistics-032921-013738
• 发表时间: 2023
• 期刊: Annual Review of Statistics and Its Application
• 影响因子: 7.9
• 作者: Nalisnick, Eric;Smyth, Padhraic;Tran, Dustin
• 通讯作者: Tran, Dustin

AI-Assisted Decision-making: a Cognitive Modeling Approach to Infer Latent Reliance Strategies

人工智能辅助决策：推断潜在依赖策略的认知建模方法

• DOI: 10.1007/s42113-022-00157-y
• 发表时间: 2022
• 期刊: Computational Brain & Behavior
• 作者: Tejeda, Heliodoro;Kumar, Aakriti;Smyth, Padhraic;Steyvers, Mark
• 通讯作者: Steyvers, Mark

Explaining Algorithm Aversion with Metacognitive Bandits

用元认知强盗解释算法厌恶

• 发表时间: 2021
• 期刊: Proceedings of the Annual Meeting of the Cognitive Science Society
• 作者: Kumar, A.;Patel, T.;Benjamin, A.;Steyvers, M.
• 通讯作者: Steyvers, M.

Combining human predictions with model probabilities via confusion matrices and calibration

通过混淆矩阵和校准将人类预测与模型概率相结合

• 发表时间: 2021
• 期刊: Advances in Neural Information Processing Systems (NeurIPS 2021
• 作者: Kerrigan, Gavin;Smyth, Padhraic;Steyvers, Mark
• 通讯作者: Steyvers, Mark