Capturing Humans’ Mental Models of AI: An Item Response Theory Approach
Capturing Humans’ Mental Models of AI: An Item Response Theory Approach
复制标题
DOI：
10.1145/3593013.3594111
复制
发表时间：
2023-05
期刊：
影响因子：
--
通讯作者：
M. Kelly;Aakriti Kumar;Padhraic Smyth;M. Steyvers
中科院分区：
文献类型：
--
作者：
M. Kelly;Aakriti Kumar;Padhraic Smyth;M. Steyvers
提高我们对人类如何看待AI队友的理解是我们对人类-AI团队的一般理解的重要基础。扩展认知科学的相关工作，我们提出了一个基于项目反应理论的框架来建模这些感知。我们将这个框架应用于现实世界的实验中，在这个实验中，每个参与者都与另一个人或人工智能代理一起在问答环境中工作，反复评估他们队友的表现。使用这些实验数据，我们展示了使用我们的框架来测试人们对AI代理和其他人的看法的研究问题。我们将人工智能队友的心理模型与人类队友的心理模型进行了对比，因为我们描述了这些心理模型的维度，它们随时间的发展以及参与者自我感知的影响。我们的研究结果表明，人们期望AI代理的表现平均比其他人的表现要好得多，不同类型的问题之间的差异较小。最后，我们讨论了这些发现对人类与人工智能互动的影响。
Improving our understanding of how humans perceive AI teammates is an important foundation for our general understanding of human-AI teams. Extending relevant work from cognitive science, we propose a framework based on item response theory for modeling these perceptions. We apply this framework to real-world experiments, in which each participant works alongside another person or an AI agent in a question-answering setting, repeatedly assessing their teammate’s performance. Using this experimental data, we demonstrate the use of our framework for testing research questions about people’s perceptions of both AI agents and other people. We contrast mental models of AI teammates with those of human teammates as we characterize the dimensionality of these mental models, their development over time, and the influence of the participants’ own self-perception. Our results indicate that people expect AI agents’ performance to be significantly better on average than the performance of other humans, with less variation across different types of problems. We conclude with a discussion of the implications of these findings for human-AI interaction.