ITR: (NHS) - (dmc+sim): Computational Audio-Visual Scene Analysis (CAVSA)

ITR：(NHS) - (dmc sim)：计算视听场景分析 (CAVSA)

• 批准号: 0426627
• 负责人: Thomas Huang
• 金额: $ 60万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2004
• 资助国家: 美国
• 起止时间: 2004-09-15 至 2008-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0426627&HistoricalAwards=false
• 关键词: ITR; NHS; dmc+sim; Computational; Audio

Abstract 0426627Huang, ThomasU of IllAudiovisual Scene AnalysisIn many applications such as Homeland Security, video surveillance is a crucial component. An important task is to extract moving objects (vehicles, people) and to separate the individual sounds (car noise, speech). But this is done usually separately, the former from the image sequences alone, and the latter from the audio (of the sound mixture). However, the two are closely coupled. We aim to do the two jointly, and term the task: Audiovisual Scene Analysis. The tool we shall use is Probabilistic Graphical Models, esp. Generative Models. A generative model includes hidden variables which affect the observed data (in this case, video with the associated audio).Using the Expectation-Maximization algorithms, these parameters can be estimated. The challenge is to come up with appropriate models such that the parameter estimation will lead to audiovisual object separation. Once the audiovisual objects in a video are separated, then techniques such as face recognition and speech analysis could be used to detect objects and events of importance to specific applications.

在国土安全等众多应用中，视频监控是一个至关重要的组成部分。一个重要的任务是提取运动物体（车辆、人）和分离单个声音（汽车噪音、语音）。但这通常是单独完成的，前者仅来自图像序列，后者来自音频（声音混合）。然而，这两者是紧密相连的。我们的目标是将两者结合起来，并将任务命名为：视听场景分析。我们将使用的工具是概率图形模型，特别是生成模型。生成模型包含影响观察数据的隐藏变量（在本例中，是带有相关音频的视频）。使用期望最大化算法，可以估计这些参数。挑战是提出适当的模型，使参数估计将导致视听对象分离。一旦视频中的视听对象被分离，那么人脸识别和语音分析等技术就可以用来检测对特定应用重要的对象和事件。