Construction of On-line Multimodal Dictionary for Automatic Human Behavior Understanding

我们研究了人类行为各个方面的描述，并以机器可读的格式积累了实际数据。本研究的内容包括：1）各种情况下的人的行为描述和积累的研究，以及原型数据库的开发; 2）多个移动摄像机的人的运动观察; 3）多个摄像机的视图选择和编辑。关于第一个主题，我们收集了各种记录：由磁运动捕捉系统测量的身体运动，来自多个摄像机的瞄准用于理解人类运动的重要部分的视频，音频，语音转录，和人类的动作诠释。它们通过它们的发生时间相互关联。我们可以通过参考这些数据来考察不同的模态在真实的情境中是如何相互配合、相互补充的。由于人类的行为严重依赖于他们所处的环境和个人特征，因此，在不同的情况下进行这种检查对于理解人类行为是必不可少的。多个移动摄像机在不同的位置。体位是针对人体的重要部位和身体周围的东西，如脸，手，桌子上的物体等。这些记录不仅有助于我们研究人类如何移动，而且还有助于我们研究这些重要部分在实际通信中是如何被看到的，以及它们携带了什么信息。在第三个主题中，我们研究了一种新的方法，通过使用上述系统获取的数据来选择观察和记录人类行为的时机和焦点。本研究的目的是利用人类运动的理解，通过选择摄像机和帧来生成可理解的报告或演讲或讲座摘要。

We investigated description for various aspects on human behaviors, and accumulated actual data in a machine readable format. This research contains, 1)a study for the human behavior description and accumulation in various situations, and development of a prototype database, 2)human motion observation by multiple moving cameras, 3)view sclection and editing from the views by multiple cameras.On the first topic, we gathered various records : body movements measured by a magnetic motion capture system, videos from multiple cameras aiming at important portions for understanding human motions, audio, speech transcripts, and human movements interprctation by human. They are related to each other by their occurrence time. We can examine, by referring the data, how the different modalities are cooperatively and complementarily used in real situations. This cxamination over different situations is essential for understanding human behaviors, since they are heavily dependent on their contexts and personal charactcristics.On the second topic, we developed a human behavior tracking system with multple cameras. Multiple moving cameras at different. positions aim at important portions on a human body and something around the body, such as face, hands, or objects on a desk. This record help us to investigate not only how humans moved but also how those important portions are seen in actual communications and what information they carry.On the third topic, we investigated a new method to choose the timing, focused point for observing and recording the human behaviors by using the data taken by the above system. This recarch aims at generation of comprehensible report or summary of presentations or lectures by selecting the cameras and frames by using human motion understanding.