CAREER: Collaboratively Perceiving, Comprehending, and Projecting into the Future: Supporting Team Situational Awareness with Adaptive Multimodal Displays

特别是在数据丰富和快速变化的环境中，有效的团队需要向成员提供必要的信息，以提高他们对自己、队友和整个团队当前状况的认识。然而，在这样的团队中，注意力需求很高，这就提出了一个问题：如何既监控这些注意力需求，又开发出一种系统，不仅通过经常过载的视觉显示，还通过包括触觉和声音在内的其他感官，自适应地提供所需的信息。大多数关于情景感知的自适应多模态接口的现有工作都集中在个体上；该项目将解决如何为团队完成这项工作，使用无人机（UAV）搜索和救援作为其主要领域。这包括开发连接个人和团队层面态势感知的概念模型，使用眼睛注视数据实时评估态势感知和工作量的算法，以及通过最有效的模式自适应地向最合适的团队成员呈现信息的多模式显示指南。这项工作将从根本上推进对理解和设计的研究，以支持团队交互，导致各种安全关键领域的实际改进。该项目还具有重要的教育组成部分，为研究生和本科生提供研究机会，并开展旨在拓展和扩大STEM学科参与的设计活动，包括工作站设计，以支持制造业背景下的残疾人团队。这项研究工作主要有两个重点。第一个涉及通过一项研究收集基线数据，其中对新手进行培训，使用标准的视觉聚焦界面执行模拟无人机搜索和救援任务；该团队将利用现有的有效调查、目光数据、团队互动数据和成员特征收集态势感知（SA）评估。这些数据将用于构建两个主要模型。第一个模型通过对观察记录和音频数据的定性分析，以及对参与者的焦点小组访谈，将团队动态和个体成员特征与SA和绩效水平联系起来。第二个是一个定量模型，它试图使用眼睛注视数据来预测SA，使用眼睛注视数据的因素分析和团队及其成员如何在界面元素和任务之间转换视觉注意力的马尔可夫模型来预测SA水平。这些模型将支持不引人注目的SA评估，避免现有调查所造成的中断，并且对于开发自适应多模式接口（项目的另一个主要推力）是必要的。第二个重点将使用在第一个研究中确定的注意力和有问题的任务和上下文模型，迭代地开发一套结合视觉、音频和触觉信息通道的多模态界面试点。这些多模式界面将使用一系列类似于第一组的研究进行评估，其目标是为呈现模式和信息类型开发成本效益模型，这些模式和信息类型对团队现有视觉工作量的干扰最小，同时仍然提供提高个人和团队SA的信息。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Especially in data-rich and rapidly changing environments, effective teams need to give members the information needed to develop awareness of their own, their teammates', and the overall team's current situation. However, attentional demands are high on such teams, raising questions of how to both monitor those attentional demands and develop systems that adaptively provide needed information not just through visual displays that are often overloaded, but through other senses including touch and sound. Most existing work on adaptive multimodal interfaces for situational awareness focuses on individuals; this project will address how to do this work for teams, using unmanned aerial vehicle (UAV) search and rescue as its primary domain. This includes developing conceptual models that connect individual and team-level situational awareness, algorithms that use eye gaze data to assess both situational awareness and workload in real-time, and multimodal display guidelines that adaptively present information to the most appropriate team members through the most effective modes. This work will fundamentally advance research on understanding and designing to support team interaction, leading to practical improvements in a variety of safety-critical domains. The project also has a significant educational component, providing research opportunities for both graduate and undergraduate students and conducting design activities aimed at outreach and broadening participation in STEM disciplines, including workstation design to support teams of people with disabilities in manufacturing contexts.The research work has two main thrusts. The first involves collecting baseline data through a study where pairs of novices are trained to carry out simulated UAV search and rescue tasks using a standard visually-focused interface; the team will collect situational awareness (SA) assessments using existing validated surveys, eye gaze data, and team interaction data and member characteristics. This data will be used to build two main models. The first is a model that relates team dynamics and individual member characteristics with levels of SA and performance, using qualitative analysis of recorded observational and audio data, along with focus group interviews with participants. The second is a quantitative model that attempts to predict SA using eye gaze data, using both a factor analysis of eye gaze data and Markovian models of how teams and their members transition their visual attention between interface elements and tasks to predict levels of SA. These models will support unobtrusive assessments of SA that avoid the interruptions imposed by existing surveys and are necessary for developing the adaptive multimodal interfaces that are the other main thrust of the project. This second thrust will use the models of attention and problematic tasks and contexts identified in the first study to iteratively develop a pilot suite of multimodal interfaces that combine visual, audio, and tactile information channels. These multimodal interfaces will be evaluated using a series of studies similar to the first set, with the goal of developing cost-benefit models for presentation modes and types of information that minimally interfere with teams' existing visual workload while still providing information that raises individual and team SA.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.