HCC-Medium: Collaborative Research: Multimodal Capture of Teamwork in Collocated Collaboration

在动态、高风险的情况下，设计和使用信息系统以支持并置团队的协作活动仍然是一项挑战。该项目将开发新的方法，以便在目前依赖人类观察、言语交流和集体记忆的环境中更有效地捕捉和交流这一活动。更高效的团队协作捕获流程将支持更大规模的收集(支持对改进培训和技术设计至关重要的回溯性分析)和同期收集(向员工提供实时反馈，以帮助检测错误)。为了实现这些目标，将使用特定领域的知识和概率推理来确定工作和沟通的模式。创伤复苏的代表性领域是这项工作的理想选择，因为球员的角色和任务都定义得很好，工作流程遵循一个通用的模式，而不考虑患者-S的受伤。由于这一环境的复杂性，使用视频记录手动跟踪所有活动需要反复审查，即使对有经验的观察员来说也非常耗时。将开发一种计算机系统，使用视频分析来确定每个玩家的位置，使用运动分析来跟踪他们的动作，并针对有限的词典进行语音识别，以识别他们的交流。使用这些输入，将构建一个概率推理模型，该模型将来自环境的数据与特定于领域的团队工作模型相关联。复苏事件的标记记录将在事件期间和事件后实时提供用于分析。这项工作的科学意义在于需要标记这些视频观测。无论是在监控应用、工作场合还是其他用途中，许多形式的视频都具有重复行为。在所有这种情况下，将语法应用于视频，并将动作和声音与该语法匹配，有可能极大地简化工作分析，这是开发计算机支持复杂、高风险人类活动的关键阶段。所提出的方法将开发新的算法和方法，用于：(I)在拥挤的协作环境中跟踪人和资源；(Ii)基于融合来自多模式传感器的不可靠数据和被记录的过程的模型来识别人类活动；以及(Iii)基于相互交互的活动和事件检测的异类技术(隐马尔可夫模型、贝叶斯网和Petri网)，对不同时间尺度上的人类活动进行推理。此外，这些方法将在目前使用有限信息技术的临床环境中开发和评估。这项工作还将为在创伤复苏和相关医疗领域等环境中实施决策辅助奠定基础，这些环境缺乏有效的方法来跟踪团队合作。创伤护理是一项重大的卫生保健危机，复苏过程中的任何改进都将拯救生命。

The design and use of information systems to support the collaborative activity of collocated teams in dynamic, high-risk scenarios remains a challenge. This project will develop novel methods to more efficiently capture and communicate this activity in environments that currently rely on human observation, verbal communication, and collective memory. More efficient teamwork capture processes will enable both larger-scale collection (which supports retrospective analysis that is critical for improved training and technology design) and contemporaneous collection (which provides real-time feedback to workers to assist in error detection). To achieve these goals, domain-specific knowledge and probabilistic reasoning will be used to identify patterns of work and communication. The representative domain of trauma resuscitation is ideal for this work since the roles and tasks of players are well-defined and the flow of work follows a general schema regardless of the patient?s injuries. Because of the complexity of this environment, manual tracking of all activities using video recordings requires repeated review and is very time-consuming even for experienced observers. A computer system will be developed that uses video analysis to determine the location of each player, motion analysis to track their movements, and speech recognition targeted at a limited lexicon to identify their communication. Using these inputs, a probabilistic reasoning model will be constructed that correlates data from the environment with a domain-specific model of teamwork. The tagged recording of the resuscitation event will be available in real time during the event as well as post-event for analysis.The scientific importance of this work is in the need to tag these video observations. Many forms of videos are of repetitive behaviors, whether in surveillance applications, work situations, or other uses. In all such cases, applying a grammar to the video, and matching actions and sounds to that grammar, has the possibility of greatly simplifying work analysis, which is the critical phase in the development computer support for complex, high-risk human activities.The proposed approach will develop novel algorithms and methods for: (i) person and resource tracking in crowded collaborative environments; (ii) recognition of human activity based on fusion of unreliable data from multimodal sensors and a model of the process being recorded; and (iii) reasoning about human activities at different time scales based on heterogeneous technologies (Hidden Markov Models, Bayesian Nets, and Petri Nets) that mutually interact for activity and event detection. Moreover, the methods will be developed and evaluated in a clinical environment that currently uses limited information technology.Broader Impacts. This work will also provide the foundation for implementing decision aids in environments such as trauma resuscitation and related medical domains that lack effective methods for instrumented tracking of teamwork. Trauma care is a significant health care crisis and any improvements in resuscitation processes will save lives.