Construction of On-line Multimodal Dictionary for Automatic Human Behavior Understanding

我们研究了对人类行为的各个方面的描述，并以机器可读的格式积累了实际数据。研究内容包括：1)研究人体在不同情况下的行为描述和积累，以及原型数据库的开发；2)多运动摄像机对人体运动的观察；3)多摄像机对视角的选择和编辑。对于第一个主题，我们收集了各种记录：磁力运动捕捉系统测量的身体运动，多个摄像机针对人体运动理解的重要部位的视频，音频，语音记录，以及人类对人体运动的解释。它们的发生时间是相互关联的。通过参考数据，我们可以检查不同的模式是如何在实际情况下协同和互补使用的。这种对不同场景的跟踪对于理解人类的行为是至关重要的，因为人类的行为严重依赖于他们所处的环境和个人特征。不同位置的多个移动摄像机。体位瞄准人体的重要部位和身体周围的东西，如脸、手或桌子上的物体。对于第三个主题，我们研究了一种新的方法，利用上述系统采集的数据来选择观察和记录人类行为的时机和焦点。这项研究的目的是通过使用人体运动理解来选择摄像机和帧来生成可理解的报告或演示文稿或讲座的摘要。

We investigated description for various aspects on human behaviors, and accumulated actual data in a machine readable format. This research contains, 1)a study for the human behavior description and accumulation in various situations, and development of a prototype database, 2)human motion observation by multiple moving cameras, 3)view sclection and editing from the views by multiple cameras.On the first topic, we gathered various records : body movements measured by a magnetic motion capture system, videos from multiple cameras aiming at important portions for understanding human motions, audio, speech transcripts, and human movements interprctation by human. They are related to each other by their occurrence time. We can examine, by referring the data, how the different modalities are cooperatively and complementarily used in real situations. This cxamination over different situations is essential for understanding human behaviors, since they are heavily dependent on their contexts and personal charactcristics.On the second topic, we developed a human behavior tracking system with multple cameras. Multiple moving cameras at different. positions aim at important portions on a human body and something around the body, such as face, hands, or objects on a desk. This record help us to investigate not only how humans moved but also how those important portions are seen in actual communications and what information they carry.On the third topic, we investigated a new method to choose the timing, focused point for observing and recording the human behaviors by using the data taken by the above system. This recarch aims at generation of comprehensible report or summary of presentations or lectures by selecting the cameras and frames by using human motion understanding.