Doctoral Dissertation Research in DRMS: Connecting Artificial Intelligence Literacy and Human-AI Decision Making Outcomes in Organizational Hiring

该奖项全部或部分由2021年美国救援计划法案（公法117-2）资助。组织越来越多地将基于人工智能（AI）的技术纳入决策过程。例如，招聘团队可以使用基于人工智能的工具来分析申请数据，如简历和面试记录，以提供招聘建议。基于人工智能的算法可以处理大量数据，并生成建议，为过去仅由人类决策者做出的决策提供信息。尽管基于人工智能的技术得到了广泛使用，但这些技术的日常非专家用户可能没有足够的人工智能知识或人工智能素养，无法使用基于人工智能的建议做出公平的决策。这个博士论文研究改进补助金（DDRIG）研究了当决策者的AI素养不同时，从基于AI的来源获得建议是否会影响结果。该项目有两个主要目的：1）在决策背景下开发一种衡量人们人工智能素养的方法，2）在招聘场景实验中测试这一方法。这项研究支持NSF的使命，即通过提供支持进一步研究人类与人工智能互动的工具来促进科学进步，随着组织继续将新的基于人工智能的技术引入工作场所，这一主题变得更加相关。这项工作可以帮助人们，无论技术背景如何，成为更明智和深思熟虑的人工智能决策者。这项研究的结果提出了一种经过验证的人工智能素养衡量标准，研究人员、个人和组织可以用来衡量个人对基于人工智能的技术的总体理解，并确定可能影响他们如何使用基于人工智能的信息做出决策的潜在知识差距。这项研究还为开发教育资源提供了信息，这些资源可以帮助求职者在基于人工智能的招聘过程中导航。第一阶段涉及规模开发工作，使用过去的研究和试点访谈数据来开发和测试人工智能素养措施的规模项目。为了确定人们的人工智能素养是否在人们如何使用基于人工智能的信息进行招聘建议方面发挥重要作用，第二阶段涉及招聘场景实验。在实验中，参与者评估模拟工作申请，并使用二次评估的输入来决定申请人是否应该在招聘过程中继续前进，参与者的人工智能素养（由第一阶段开发的量表测量）被用作控制变量。这项调查的结果可能有助于各利益相关者更好地了解人们对人工智能的理解如何影响决策结果。这些发现有可能为人工智能培训和教育工作做出贡献。此外，该实验是人工智能素养衡量标准的首次应用，它催化了未来探索人工智能素养与人类-人工智能决策之间关系的研究。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Organizations are increasingly incorporating artificial intelligence (AI)-based technologies into decision-making processes. For example, hiring teams may use AI-based tools that analyze application data, such as resumes and interview recordings, to provide hiring recommendations. AI-based algorithms can process large amounts of data and generate recommendations that inform decisions that used to be made solely by human decision-makers. Despite the widespread use of AI-based technologies, everyday non-expert users of these technologies may not have sufficient knowledge about AI, or AI literacy, to make fair decisions using AI-based recommendations. This Doctoral Dissertation Research Improvement Grant (DDRIG) examines whether receiving recommendations from an AI-based source impacts outcomes when decision-makers vary in their AI literacy. This project has two primary purposes: 1) to develop a measure for people's AI literacy within the context of decision-making, and 2) to test this measure in a hiring scenario experiment. This research supports NSF’s mission to promote the progress of science by contributing tools that support further research on human-AI interaction, a subject becoming more relevant as organizations continue to introduce new AI-based technologies into the workplace. This work helps people, regardless of technical background, become better informed and thoughtful human-AI decision-makers. The findings from this research advance a validated measure for AI literacy that researchers, individuals, and organizations can use to measure individuals' general understanding of AI-based technologies and identify potential gaps in knowledge that can impact how they use AI-based information to make decisions. This research also informs the development of educational resources that help job seekers navigate the AI-based hiring process.The research entails two phases. Phase I involves a scale development effort that uses past research and pilot interview data to develop and test scale items for an AI literacy measure. To determine whether people's AI literacy plays a significant role in how people use AI-based information to make hiring recommendations, phase II involves a hiring scenario experiment. In the experiment, participants evaluate a mock job application and use input from a secondary evaluation to decide whether the applicant should move forward in the hiring process, and participants' AI literacy (as measured by the scale developed in phase I) is used as a control variable. The results from this investigation may help various stakeholders better understand how people's understanding of AI influences decision-making outcomes. The findings have the potential to contribute to work on AI training and education at large. Furthermore, the experiment is the first application of the AI literacy measure that catalyzes future research exploring the relationship between AI literacy and human-AI decision-making.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.