RAPID/Collaborative Research: Human-AI Teaming for Big Data Analytics to Enhance Response to the COVID-19 Pandemic

社交媒体数据可以提供重要的线索和当地知识，帮助应急管理人员和响应人员更好地理解和捕捉许多灾难的演变性质。然而，只有人类无法掌握社交媒体产生的海量数据，因此计算机被用来提供帮助。目前，人们对如何利用人类和机器在一起工作时的技能(人机协作)来识别社交媒体数据中的有意义模式知之甚少。因此，这个快速反应研究(RAPID)项目寻求解决的基本问题是1)了解人类数字志愿者在将社交媒体数据快速转换为机器(人工智能算法)可以理解的结构化代码时做出实时决策的过程，以及2)使用这些知识来改进人机协作。该项目通过揭示人类和机器在不断演变的灾难中合作理解社交媒体模式时带来的独特能力，推动了这一领域的发展。它通过向不同的学生提供研究经验，以及生成对教学团队合作、社交媒体分析和人机合作的跨学科课程有用的数据，来支持教育和多样性。最后，这些发现可以帮助应急经理更好地培训志愿者，这些志愿者利用他们对当地知识和建筑环境的理解，梳理社交媒体，帮助机器在数据中看到新的模式。因此，该项目支持NSF的使命，即通过阐明人类和计算机带来的独特价值，促进科学进步，促进国家的健康、繁荣和福利，这些价值可以在灾难中导致更好的决策。本研究的目的是更好地理解人类注释者在不同环境约束下做出的实时决策，以及这些决策如何对人工智能(AI)模型的学习做出贡献。在时间限制和信息过载的情况下，人类的决策能力是有限的；然而，人类仍然具有独特的能力，能够理解机器无法识别的对建成环境中结构的上下文引用。例如，在缺乏建筑环境知识的人工智能系统上，这条推文的意义可能会被遗漏。这条推文的意思是，这家名为Memorial的医院已经没有病人的床位了。这个例子展示了人类在人工智能团队环境中提供的价值。本研究专注于从各种社交媒体来源获取短暂的数据，我们的两项研究工作包括：1)使用有声思考和认知访谈策略对社区应急小组(CERT)志愿者和一名经理(该项目的合作者)进行在线观察，以揭示用于为注释任务做出编码决策的实时心理模型；以及2)对主动(机器)学习范式的不同采样算法进行实证分析，以开发不同背景下影响人类注释决策质量的机器错误类型。这项研究将产生设计指导方针，弥合人工智能模型用于实时数据处理的机制与人类用户与人工智能模型合作所贡献的对上下文的理解之间的差距。利用人类决策理论和人工智能如何运作的知识，该项目提供了1)人类如何理解、处理和解释社交媒体消息的实时中途检查，以及2)如何改进人工智能算法以优化主动学习范式。这一理解将提供一个理论框架，使未来的研究能够开发协议，通过使用动机和信息理论等概念来优化人类-人工智能合作。这项工作可以帮助应急管理人员对他们的CERT志愿者和其他注释员进行更好的培训，并提供更明确的指导方针，说明如何传达人类为人工智能系统的注解过程带来的独特价值。我们的协议和对人类如何与人工智能系统交互的深入理解将有助于全球卫生组织、地方和州一级的灾难决策者，并为美国庞大的CERT网络提供方向。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Social media data can provide important clues and local knowledge that can help emergency managers and responders better comprehend and capture the evolving nature of many disasters. Yet humans alone cannot grasp the vast data generated by social media, so computers are used to assist. Very little is currently known about how to leverage the skills of humans and machines when they work together (human-machine teaming) to identify meaningful patterns in social media data. Therefore, the fundamental issues this Rapid Response Research (RAPID) project seeks to address are 1) understanding the process of real-time decisions that human digital volunteers make when they rapidly convert social media data into structured codes the machine (Artificial Intelligence algorithms) can understand, and 2) using this knowledge to improve human-machine teaming. This project advances the field by revealing the unique abilities that both humans and machines bring when working together to comprehend social media patterns during an evolving disaster. It supports education and diversity by providing research experiences to diverse students, as well as generating data useful for interdisciplinary courses teaching teamwork, social media analysis, and human-machine teaming. Finally, the findings can help emergency managers better train their volunteers who comb through social media using their understanding of the local knowledge and built environment to help machines see new patterns in data. Hence, this project supports NSF's mission to promote the progress of science and to advance the nation's health, prosperity, and welfare by articulating the unique value that both humans and computers bring that can lead to better decisions during disasters. The goal of this research is to better understand the real-time decisions that human annotators make under different environmental constraints, and how those contribute to the learning of Artificial Intelligence (AI) models. Under time constraints and information overload, human decision-making capabilities are limited; yet, humans still have a unique ability to understand the contextual references to the structures in the built environment that machines cannot recognize. For example, the meaning of the tweet, “Memorial is overloaded,” -- which means the hospital, called Memorial, is out of beds for patients —- can be lost on AI systems that lack the knowledge of the built environment. This example demonstrates the value that humans in the loop offer in a human-AI teaming context. This research focuses on capturing the ephemeral data from a variety of social media sources and our two research thrusts include: 1) online observations of Community Emergency Response Team (CERT) volunteers and a manager (a collaborator on this project) using think-aloud and cognitive interviewing strategies to reveal the real-time mental models used to make coding decisions for annotation tasks; and 2) an empirical analysis of different sampling algorithms for active (machine) learning paradigms to develop a typology of machine errors under diverse contexts that affect the quality of human decision making for annotation. This research will generate design guidelines that bridge the gap between the mechanisms used for real-time data processing with AI models and the understanding of context contributed by a human user teaming with the AI models. Using theories of human decision-making combined with knowledge of how AI functions, this project provides a real-time, mid-disaster examination of 1) how humans understand, process, and interpret social media messages, and 2) how to refine AI algorithms to optimize active learning paradigm. This understanding will provide a theoretical framework enabling future research to develop protocols to optimize human-AI teaming by using concepts such as motivation and information theory. This work can help emergency managers conduct better training of their CERT volunteers and other annotators and provide clearer guidelines for how to communicate the unique value that humans bring to the annotation process for AI systems. Both our protocols and developed understanding of how humans interact with AI systems will be helpful for global health organizations, local and state-level disaster decision-makers, as well as provide direction for the vast CERT network in the United States.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.