HCC-Medium: Collaborative Research: Multimodal Capture of Teamwork in Collocated Collaboration

HCC-Medium：协作研究：协同协作中团队合作的多模式捕获

• 批准号: 0929705
• 负责人: Randall Burd
• 金额: $ 15万
• 依托单位: Children's Research Institute
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2008
• 资助国家: 美国
• 起止时间: 2008-12-15 至 2011-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0929705&HistoricalAwards=false
• 关键词: HCC; Medium; Collaborative; Research; Multimodal

The design and use of information systems to support the collaborative activity of collocated teams in dynamic, high-risk scenarios remains a challenge. This project will develop novel methods to more efficiently capture and communicate this activity in environments that currently rely on human observation, verbal communication, and collective memory. More efficient teamwork capture processes will enable both larger-scale collection (which supports retrospective analysis that is critical for improved training and technology design) and contemporaneous collection (which provides real-time feedback to workers to assist in error detection). To achieve these goals, domain-specific knowledge and probabilistic reasoning will be used to identify patterns of work and communication. The representative domain of trauma resuscitation is ideal for this work since the roles and tasks of players are well-defined and the flow of work follows a general schema regardless of the patient?s injuries. Because of the complexity of this environment, manual tracking of all activities using video recordings requires repeated review and is very time-consuming even for experienced observers. A computer system will be developed that uses video analysis to determine the location of each player, motion analysis to track their movements, and speech recognition targeted at a limited lexicon to identify their communication. Using these inputs, a probabilistic reasoning model will be constructed that correlates data from the environment with a domain-specific model of teamwork. The tagged recording of the resuscitation event will be available in real time during the event as well as post-event for analysis. The scientific importance of this work is in the need to tag these video observations. Many forms of videos are of repetitive behaviors, whether in surveillance applications, work situations, or other uses. In all such cases, applying a grammar to the video, and matching actions and sounds to that grammar, has the possibility of greatly simplifying work analysis, which is the critical phase in the development computer support for complex, high-risk human activities. The proposed approach will develop novel algorithms and methods for: (i) person and resource tracking in crowded collaborative environments; (ii) recognition of human activity based on fusion of unreliable data from multimodal sensors and a model of the process being recorded; and (iii) reasoning about human activities at different time scales based on heterogeneous technologies (Hidden Markov Models, Bayesian Nets, and Petri Nets) that mutually interact for activity and event detection. Moreover, the methods will be developed and evaluated in a clinical environment that currently uses limited information technology. Broader Impacts. This work will also provide the foundation for implementing decision aids in environments such as trauma resuscitation and related medical domains that lack effective methods for instrumented tracking of teamwork. Trauma care is a significant health care crisis and any improvements in resuscitation processes will save lives.

在动态的、高风险的情况下，设计和使用信息系统来支持协同团队的协作活动仍然是一个挑战。该项目将开发新的方法，在目前依赖人类观察、口头交流和集体记忆的环境中更有效地捕捉和交流这种活动。更有效的团队协作捕获过程将支持更大规模的收集（支持对改进培训和技术设计至关重要的回顾性分析）和同步收集（为工人提供实时反馈以协助错误检测）。为了实现这些目标，将使用特定于领域的知识和概率推理来确定工作和通信的模式。创伤复苏的代表性领域是这项工作的理想选择，因为参与者的角色和任务是明确定义的，工作流程遵循一般模式，无论患者如何？年代受伤。由于这种环境的复杂性，使用录像对所有活动进行手动跟踪需要反复审查，即使对经验丰富的观察员来说也是非常耗时的。将开发一种计算机系统，使用视频分析来确定每个球员的位置，动作分析来跟踪他们的动作，以及针对有限词汇的语音识别来识别他们的交流。使用这些输入，将构建一个概率推理模型，该模型将来自环境的数据与特定于领域的团队合作模型相关联。复苏事件的标记记录将在事件期间和事件后实时可用，以供分析。这项工作的科学重要性在于需要标记这些视频观测。许多形式的视频都是重复的行为，无论是在监控应用、工作环境还是其他用途中。在所有这些情况下，将语法应用于视频，并将动作和声音与该语法相匹配，有可能大大简化工作分析，这是开发计算机支持复杂，高风险人类活动的关键阶段。提出的方法将开发新的算法和方法：(i)在拥挤的协作环境中跟踪人员和资源；（ii）根据来自多模态传感器的不可靠数据和正在记录的过程模型的融合来识别人类活动；（iii）基于异构技术（隐马尔可夫模型、贝叶斯网和Petri网）对不同时间尺度的人类活动进行推理，这些技术相互作用，用于活动和事件检测。此外，这些方法将在目前使用有限信息技术的临床环境中开发和评估。更广泛的影响。这项工作还将为在创伤复苏和相关医疗领域中实施决策辅助提供基础，这些领域缺乏有效的团队合作仪器跟踪方法。创伤护理是一项重大的卫生保健危机，复苏过程的任何改进都将挽救生命。