CAREER: Collaboratively Perceiving, Comprehending, and Projecting into the Future: Supporting Team Situational Awareness with Adaptive Multimodal Displays

特别是在数据丰富和快速变化的环境中，有效的团队需要为成员提供所需的信息，以提高对自己、队友和整个团队现状的认识。 然而，对这些团队的注意力要求很高，这就提出了一个问题，即如何既监控这些注意力需求，又开发出一种系统，不仅通过经常超载的视觉显示，而且通过包括触觉和声音在内的其他感官，自适应地提供所需的信息。 大多数现有的工作，自适应多模态界面的态势感知集中在个人，这个项目将解决如何做这项工作的团队，使用无人驾驶航空器（UAV）的搜索和救援作为其主要领域。 这包括开发连接个人和团队级别态势感知的概念模型，使用眼睛凝视数据实时评估态势感知和工作量的算法，以及通过最有效的模式自适应地向最合适的团队成员呈现信息的多模式显示指南。 这项工作将从根本上推进对理解和设计的研究，以支持团队互动，从而在各种安全关键领域实现实际改进。 该项目还具有重要的教育组成部分，为研究生和本科生提供研究机会，并开展旨在推广和扩大STEM学科参与的设计活动，包括工作站设计，以支持制造环境中的残疾人团队。研究工作有两个主要重点。 第一个是通过一项研究收集基线数据，在这项研究中，成对的新手接受培训，使用标准的视觉聚焦界面执行模拟的无人机搜索和救援任务;该团队将使用现有的有效调查、眼睛注视数据、团队互动数据和成员特征收集态势感知（SA）评估。 这些数据将用于构建两个主要模型。 首先是一个模型，涉及团队的动态和个人成员的特点与水平的SA和性能，使用定性分析记录的观察和音频数据，沿着与焦点小组访谈的参与者。 第二个是一个定量模型，试图预测SA使用眼睛凝视数据，使用眼睛凝视数据的因素分析和马尔可夫模型的团队及其成员如何过渡界面元素和任务之间的视觉注意力，以预测SA的水平。 这些模型将支持对SA的不引人注目的评估，避免现有调查造成的中断，并且对于开发作为该项目另一个主要目标的自适应多模式界面是必要的。 这第二个推力将使用模型的注意力和有问题的任务和环境中确定的第一项研究，迭代开发一个试点套件的多模式界面，结合联合收割机的视觉，听觉和触觉信息渠道。 这些多模式界面将使用一系列类似于第一组的研究进行评估，以开发成本为目标-该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识产权进行评估来支持。优点和更广泛的影响审查标准。

Especially in data-rich and rapidly changing environments, effective teams need to give members the information needed to develop awareness of their own, their teammates', and the overall team's current situation. However, attentional demands are high on such teams, raising questions of how to both monitor those attentional demands and develop systems that adaptively provide needed information not just through visual displays that are often overloaded, but through other senses including touch and sound. Most existing work on adaptive multimodal interfaces for situational awareness focuses on individuals; this project will address how to do this work for teams, using unmanned aerial vehicle (UAV) search and rescue as its primary domain. This includes developing conceptual models that connect individual and team-level situational awareness, algorithms that use eye gaze data to assess both situational awareness and workload in real-time, and multimodal display guidelines that adaptively present information to the most appropriate team members through the most effective modes. This work will fundamentally advance research on understanding and designing to support team interaction, leading to practical improvements in a variety of safety-critical domains. The project also has a significant educational component, providing research opportunities for both graduate and undergraduate students and conducting design activities aimed at outreach and broadening participation in STEM disciplines, including workstation design to support teams of people with disabilities in manufacturing contexts.The research work has two main thrusts. The first involves collecting baseline data through a study where pairs of novices are trained to carry out simulated UAV search and rescue tasks using a standard visually-focused interface; the team will collect situational awareness (SA) assessments using existing validated surveys, eye gaze data, and team interaction data and member characteristics. This data will be used to build two main models. The first is a model that relates team dynamics and individual member characteristics with levels of SA and performance, using qualitative analysis of recorded observational and audio data, along with focus group interviews with participants. The second is a quantitative model that attempts to predict SA using eye gaze data, using both a factor analysis of eye gaze data and Markovian models of how teams and their members transition their visual attention between interface elements and tasks to predict levels of SA. These models will support unobtrusive assessments of SA that avoid the interruptions imposed by existing surveys and are necessary for developing the adaptive multimodal interfaces that are the other main thrust of the project. This second thrust will use the models of attention and problematic tasks and contexts identified in the first study to iteratively develop a pilot suite of multimodal interfaces that combine visual, audio, and tactile information channels. These multimodal interfaces will be evaluated using a series of studies similar to the first set, with the goal of developing cost-benefit models for presentation modes and types of information that minimally interfere with teams' existing visual workload while still providing information that raises individual and team SA.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.