HCC-Medium: Collaborative Research: Multimodal Capture of Teamwork in Collocated Collaboration

设计和使用信息系统，以支持在动态，高风险的情况下协同工作的团队的协作活动仍然是一个挑战。该项目将开发新的方法，以更有效地捕捉和交流这种活动的环境中，目前依赖于人类的观察，口头交流和集体记忆。更有效的团队协作捕获流程将实现更大规模的收集（支持对改进培训和技术设计至关重要的回顾性分析）和同期收集（向工作人员提供实时反馈，以协助发现错误）。为了实现这些目标，将使用特定领域的知识和概率推理来确定工作和沟通的模式。创伤复苏的代表性领域是这项工作的理想选择，因为球员的角色和任务是明确定义的，工作流程遵循一个一般的模式，无论病人？的伤害。由于这一环境的复杂性，使用视频记录对所有活动进行手动跟踪需要反复审查，即使对于有经验的观察者来说也非常耗时。将开发一个计算机系统，该系统使用视频分析来确定每个玩家的位置，使用运动分析来跟踪他们的运动，并使用针对有限词汇的语音识别来识别他们的通信。使用这些输入，将构建一个概率推理模型，该模型将来自环境的数据与特定领域的团队合作模型相关联。复苏事件的标记记录将在事件期间以及事件后的真实的时间内提供，以供分析。这项工作的科学重要性在于需要标记这些视频观察。许多形式的视频都是重复的行为，无论是在监控应用程序中，工作情况下，还是其他用途。在所有这些情况下，将语法应用于视频，并将动作和声音与该语法相匹配，有可能大大简化工作分析，这是开发计算机支持复杂，高风险人类活动的关键阶段。所提出的方法将开发新的算法和方法：（i）在拥挤的协作环境中的人和资源跟踪;（ii）基于多模态传感器和记录过程模型的不可靠数据的融合来识别人类活动;以及（iii）基于异构技术在不同时间尺度上推理人类活动（隐马尔可夫模型，贝叶斯网和Petri网），相互作用的活动和事件检测。此外，这些方法将在目前使用有限信息技术的临床环境中开发和评估。更广泛的影响。这项工作还将为在缺乏有效的仪器跟踪团队合作方法的环境中实施决策辅助工具（如创伤复苏和相关医疗领域）提供基础。创伤护理是一个重大的卫生保健危机，复苏过程的任何改进都将挽救生命。

The design and use of information systems to support the collaborative activity of collocated teams in dynamic, high-risk scenarios remains a challenge. This project will develop novel methods to more efficiently capture and communicate this activity in environments that currently rely on human observation, verbal communication, and collective memory. More efficient teamwork capture processes will enable both larger-scale collection (which supports retrospective analysis that is critical for improved training and technology design) and contemporaneous collection (which provides real-time feedback to workers to assist in error detection). To achieve these goals, domain-specific knowledge and probabilistic reasoning will be used to identify patterns of work and communication. The representative domain of trauma resuscitation is ideal for this work since the roles and tasks of players are well-defined and the flow of work follows a general schema regardless of the patient?s injuries. Because of the complexity of this environment, manual tracking of all activities using video recordings requires repeated review and is very time-consuming even for experienced observers. A computer system will be developed that uses video analysis to determine the location of each player, motion analysis to track their movements, and speech recognition targeted at a limited lexicon to identify their communication. Using these inputs, a probabilistic reasoning model will be constructed that correlates data from the environment with a domain-specific model of teamwork. The tagged recording of the resuscitation event will be available in real time during the event as well as post-event for analysis. The scientific importance of this work is in the need to tag these video observations. Many forms of videos are of repetitive behaviors, whether in surveillance applications, work situations, or other uses. In all such cases, applying a grammar to the video, and matching actions and sounds to that grammar, has the possibility of greatly simplifying work analysis, which is the critical phase in the development computer support for complex, high-risk human activities. The proposed approach will develop novel algorithms and methods for: (i) person and resource tracking in crowded collaborative environments; (ii) recognition of human activity based on fusion of unreliable data from multimodal sensors and a model of the process being recorded; and (iii) reasoning about human activities at different time scales based on heterogeneous technologies (Hidden Markov Models, Bayesian Nets, and Petri Nets) that mutually interact for activity and event detection. Moreover, the methods will be developed and evaluated in a clinical environment that currently uses limited information technology. Broader Impacts. This work will also provide the foundation for implementing decision aids in environments such as trauma resuscitation and related medical domains that lack effective methods for instrumented tracking of teamwork. Trauma care is a significant health care crisis and any improvements in resuscitation processes will save lives.