ITR: (NHS) - (dmc+sim): Computational Audio-Visual Scene Analysis (CAVSA)

0426627黄、托马苏的视听场景分析在国土安全部等许多应用中，视频监控是至关重要的组成部分。一项重要的任务是提取运动对象(车辆、人)，并分离出单独的声音(汽车噪声、语音)。但这通常是单独完成的，前者仅从图像序列中进行，而后者从(声音混合的)音频中进行。然而，这两者是紧密相连的。我们的目标是共同完成这两项任务，并将其命名为：视听场景分析。我们将使用的工具是概率图形模型，特别是。生成性模型。生成模型包括影响观测数据的隐藏变量(在这种情况下，视频和关联的音频)。使用期望最大化算法，可以估计这些参数。挑战在于提出合适的模型，使得参数估计将导致视听对象分离。一旦视频中的视听对象被分离，则可以使用诸如人脸识别和语音分析之类的技术来检测对特定应用重要的对象和事件。

Abstract 0426627Huang, ThomasU of IllAudiovisual Scene AnalysisIn many applications such as Homeland Security, video surveillance is a crucial component. An important task is to extract moving objects (vehicles, people) and to separate the individual sounds (car noise, speech). But this is done usually separately, the former from the image sequences alone, and the latter from the audio (of the sound mixture). However, the two are closely coupled. We aim to do the two jointly, and term the task: Audiovisual Scene Analysis. The tool we shall use is Probabilistic Graphical Models, esp. Generative Models. A generative model includes hidden variables which affect the observed data (in this case, video with the associated audio).Using the Expectation-Maximization algorithms, these parameters can be estimated. The challenge is to come up with appropriate models such that the parameter estimation will lead to audiovisual object separation. Once the audiovisual objects in a video are separated, then techniques such as face recognition and speech analysis could be used to detect objects and events of importance to specific applications.