CAREER: Collaboratively Perceiving, Comprehending, and Projecting into the Future: Supporting Team Situational Awareness with Adaptive Multimodal Displays

特别是在数据丰富和快速变化的环境中，有效的团队需要向成员提供所需的信息，以提高对自己、队友和整个团队现状的认识。然而，这类团队的注意力需求很高，这引发了一个问题，即如何既监控这些注意力需求，又开发出不仅通过经常超载的视觉显示器，而且通过包括触觉和声音在内的其他感官自适应地提供所需信息的系统。大多数现有的关于用于态势感知的自适应多模式接口的工作都集中在个人身上；这个项目将解决如何为团队做这项工作，将无人机(UAV)搜索和救援作为其主要领域。这包括开发连接个人和团队级别的态势感知的概念模型，使用眼睛凝视数据实时评估态势感知和工作量的算法，以及通过最有效的模式自适应地将信息呈现给最合适的团队成员的多模式显示指南。这项工作将从根本上促进对支持团队交互的理解和设计的研究，导致在各种安全关键领域的实际改进。该项目还具有重要的教育组成部分，为研究生和本科生提供研究机会，并开展旨在拓展和扩大STEM学科参与的设计活动，包括支持制造业中的残疾人团队的工作站设计。研究工作有两个主要方面。第一项研究涉及通过一项研究收集基线数据，在这项研究中，两对新手接受培训，使用标准的以视觉为重点的界面执行模拟无人机搜索和救援任务；团队将使用现有的经过验证的调查、眼睛凝视数据以及团队互动数据和成员特征来收集态势感知(SA)评估。这些数据将用于构建两个主要模型。第一个是一个模型，它将团队动态和个人成员特征与SA和绩效水平联系起来，使用对记录的观察和音频数据的定性分析，以及对参与者的焦点小组访谈。第二种是定量模型，它试图使用眼睛凝视数据来预测SA，既使用了眼睛凝视数据的因子分析，也使用了团队及其成员如何在界面元素和任务之间转移视觉注意力的马尔可夫模型，以预测SA的水平。这些模型将支持对SA的低调评估，避免现有调查造成的干扰，并且对于开发自适应多模式接口是必要的，而自适应多模式接口是项目的另一个主要推动力。第二个推动力将使用第一个研究中确定的注意力模型和有问题的任务和背景，反复开发一套将视觉、音频和触觉信息通道结合起来的多模式界面试点套件。这些多模式界面将使用与第一组类似的一系列研究进行评估，目标是为呈现模式和信息类型开发成本效益模型，这些模式和类型的信息最大限度地干扰团队现有的视觉工作，同时仍提供提高个人和团队SA的信息。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Especially in data-rich and rapidly changing environments, effective teams need to give members the information needed to develop awareness of their own, their teammates', and the overall team's current situation. However, attentional demands are high on such teams, raising questions of how to both monitor those attentional demands and develop systems that adaptively provide needed information not just through visual displays that are often overloaded, but through other senses including touch and sound. Most existing work on adaptive multimodal interfaces for situational awareness focuses on individuals; this project will address how to do this work for teams, using unmanned aerial vehicle (UAV) search and rescue as its primary domain. This includes developing conceptual models that connect individual and team-level situational awareness, algorithms that use eye gaze data to assess both situational awareness and workload in real-time, and multimodal display guidelines that adaptively present information to the most appropriate team members through the most effective modes. This work will fundamentally advance research on understanding and designing to support team interaction, leading to practical improvements in a variety of safety-critical domains. The project also has a significant educational component, providing research opportunities for both graduate and undergraduate students and conducting design activities aimed at outreach and broadening participation in STEM disciplines, including workstation design to support teams of people with disabilities in manufacturing contexts.The research work has two main thrusts. The first involves collecting baseline data through a study where pairs of novices are trained to carry out simulated UAV search and rescue tasks using a standard visually-focused interface; the team will collect situational awareness (SA) assessments using existing validated surveys, eye gaze data, and team interaction data and member characteristics. This data will be used to build two main models. The first is a model that relates team dynamics and individual member characteristics with levels of SA and performance, using qualitative analysis of recorded observational and audio data, along with focus group interviews with participants. The second is a quantitative model that attempts to predict SA using eye gaze data, using both a factor analysis of eye gaze data and Markovian models of how teams and their members transition their visual attention between interface elements and tasks to predict levels of SA. These models will support unobtrusive assessments of SA that avoid the interruptions imposed by existing surveys and are necessary for developing the adaptive multimodal interfaces that are the other main thrust of the project. This second thrust will use the models of attention and problematic tasks and contexts identified in the first study to iteratively develop a pilot suite of multimodal interfaces that combine visual, audio, and tactile information channels. These multimodal interfaces will be evaluated using a series of studies similar to the first set, with the goal of developing cost-benefit models for presentation modes and types of information that minimally interfere with teams' existing visual workload while still providing information that raises individual and team SA.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.