Construction of On-line Multimodal Dictionary for Automatic Human Behavior Understanding

我们研究了人类行为的各个方面的描述，并以机器可读的格式积累了实际数据。本研究包括：1)研究不同情境下的人体行为描述与积累，开发原型数据库；2)多运动摄像机对人体运动的观察；3)多摄像机对视图的选择与编辑。在第一个主题上，我们收集了各种记录：由磁性运动捕捉系统测量的身体运动，来自多个摄像机的视频，旨在理解人类运动的重要部分，音频，语音转录，以及人类对人类运动的解释。它们的发生时间是相互关联的。我们可以通过参考数据来检查不同的模式是如何在实际情况下合作和互补使用的。这种对不同情况的考察对于理解人类的行为是必不可少的，因为它们在很大程度上依赖于它们的环境和个人特征。关于第二个主题，我们开发了一个带有多个摄像头的人类行为跟踪系统。多个移动摄像机在不同的。体位的目标是人体的重要部位和身体周围的东西，如脸、手或桌子上的物体。这一记录不仅帮助我们研究人类是如何移动的，而且还帮助我们研究这些重要的部分是如何在实际交流中被看到的，以及它们携带了什么信息。在第三个主题中，我们研究了一种利用上述系统采集的数据来选择观察和记录人类行为的时间点和焦点的新方法。本研究的目的是利用人体运动理解，通过选择相机和帧，生成可理解的报告或演讲或讲座的总结。

We investigated description for various aspects on human behaviors, and accumulated actual data in a machine readable format. This research contains, 1)a study for the human behavior description and accumulation in various situations, and development of a prototype database, 2)human motion observation by multiple moving cameras, 3)view sclection and editing from the views by multiple cameras.On the first topic, we gathered various records : body movements measured by a magnetic motion capture system, videos from multiple cameras aiming at important portions for understanding human motions, audio, speech transcripts, and human movements interprctation by human. They are related to each other by their occurrence time. We can examine, by referring the data, how the different modalities are cooperatively and complementarily used in real situations. This cxamination over different situations is essential for understanding human behaviors, since they are heavily dependent on their contexts and personal charactcristics.On the second topic, we developed a human behavior tracking system with multple cameras. Multiple moving cameras at different. positions aim at important portions on a human body and something around the body, such as face, hands, or objects on a desk. This record help us to investigate not only how humans moved but also how those important portions are seen in actual communications and what information they carry.On the third topic, we investigated a new method to choose the timing, focused point for observing and recording the human behaviors by using the data taken by the above system. This recarch aims at generation of comprehensible report or summary of presentations or lectures by selecting the cameras and frames by using human motion understanding.