Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems
Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems
复制标题
DOI：
10.1145/3397481.3450639
复制
发表时间：
2021-04
期刊：
影响因子：
--
通讯作者：
Mahsan Nourani;Chiradeep Roy;Jeremy E. Block;Donald R. Honeycutt;Tahrima Rahman;E. Ragan;Vibhav Gogate
中科院分区：
文献类型：
--
作者：
Mahsan Nourani;Chiradeep Roy;Jeremy E. Block;Donald R. Honeycutt;Tahrima Rahman;E. Ragan;Vibhav Gogate
可解释的人工智能 (XAI) 方法用于为机器学习和人工智能模型带来透明度，从而改善最终用户的决策过程。虽然这些方法旨在提高人类的理解和心理模型，但认知偏差仍然会以系统设计者无法预料的方式影响用户的心理模型和决策。本文介绍了智能系统中排序效应引起的认知偏差的研究。我们进行了一项受控用户研究，以了解观察系统弱点和优势的顺序如何影响用户的心理模型、任务绩效以及对智能系统的依赖，并研究解释在解决这种偏见中的作用。在烹饪领域使用可解释的视频活动识别工具，我们要求参与者验证是否遵循了一组厨房政策，每项政策都侧重于弱点或优势。我们控制了政策的顺序和解释的存在来检验我们的假设。我们的主要发现表明，那些早期观察系统优势的人更容易出现自动化偏差，并且由于对系统的积极第一印象而犯下更多错误，同时他们建立了更准确的系统能力心理模型。另一方面，那些较早遇到弱点的人犯的错误明显更少，因为他们倾向于更多地依赖自己，同时他们也由于对模型的第一印象更负面而低估了模型的能力。我们的工作提出了强有力的发现，旨在让智能系统设计者在设计此类工具时意识到此类偏见。
EXplainable Artificial Intelligence (XAI) approaches are used to bring transparency to machine learning and artificial intelligence models, and hence, improve the decision-making process for their end-users. While these methods aim to improve human understanding and their mental models, cognitive biases can still influence a user’s mental model and decision-making in ways that system designers do not anticipate. This paper presents research on cognitive biases due to ordering effects in intelligent systems. We conducted a controlled user study to understand how the order of observing system weaknesses and strengths can affect the user’s mental model, task performance, and reliance on the intelligent system, and we investigate the role of explanations in addressing this bias. Using an explainable video activity recognition tool in the cooking domain, we asked participants to verify whether a set of kitchen policies are being followed, with each policy focusing on a weakness or a strength. We controlled the order of the policies and the presence of explanations to test our hypotheses. Our main finding shows that those who observed system strengths early-on were more prone to automation bias and made significantly more errors due to positive first impressions of the system, while they built a more accurate mental model of the system competencies. On the other hand, those who encountered weaknesses earlier made significantly fewer errors since they tended to rely more on themselves, while they also underestimated model competencies due to having a more negative first impression of the model. Our work presents strong findings that aim to make intelligent system designers aware of such biases when designing such tools.