Understanding human intention and activities for versatile real-time human-machine-interactions

在21世纪，我们的个人和社会活动是在两个不同的领域进行的：物理现实世界和网络社会。为了实现这些领域的流畅和随意的集成，我们需要一种新的交互模型，它比现有的“命令-响应模型”更进一步。在本研究中，我们提出了“人机共生系统”的概念作为下一代交互模型。人机共生系统的特点是，即使没有明确的命令，系统也能为人类工作。为了实现这样一个共生系统，需要设计实时的双边多模态信息处理机制：观察人类的行为，了解其活动和意图，并在适当的时间提供适当的信息，自愿地支持他们。为了实现人机共生系统，我们进行了以下基础研究，获得了包括理论模型和现实交互系统在内的一系列重要成果：观察人类活动（识别系统）：我们开发了一个基于实时三维运动测量的摄像机网络系统和可穿戴的主动视觉传感器系统，用于了解人类的意图和活动。吸引人（生成系统）：我们开发了一种智能桌面系统，可以根据双手操作桌面上的投影对象，在桌面板上呈现多种信息。与人交互（交互模型）：我们提出了一个动态事件表示模型，我们称之为“混合动力系统”。我们将该模型应用于细粒度面部表情分类，并分析了嘴部运动与语音的同步机制，验证了该模型的有效性。我们还通过对日本单口喜剧（漫仔）的分析，解释了实现流畅互动的基本原则。

In the 21st century, our personal and social activities are conducted in two different domains: physical real world and cyber network society. For realizing smooth and casual integration of these domains, we need a novel interaction model that goes a step further beyond the existing "command-and-response model."In this research, we proposed a concept of "man-machine symbiotic systems" as the next generation interaction model. Man-machine symbiotic systems are characterized by such a feature that the systems work for humans even if they are not explicitly ordered. For realizing such a symbiotic system, it is required to design real-time bi-lateral multimodal information processing mechanisms: observing human behaviors to understand their activities and intentions, and supporting them voluntarily by presenting appropriate information at an appropriate timing.To realize man-machine symbiotic systems, we conducted the following fundamental researches and obtained a number of important results including theoretical models and real world interaction systems:1. Observing Human Activities (recognition system): We developed a camera-network system and wearable active vision sensor systems for understanding human intentions and activities based on realtime 3D motion measurement.2. Attracting Human (generation system): We developed an intelligent desk system, which presents versatile information on a desk top plate according to two-handed manipulations of projected objects on the plate.3. Interacting with Human (interaction model): We proposed a dynamic event representation model which we refer to as a "hybrid dynamical system". We verified the effectiveness of the model by applying it to classify fine-grained facial expression, and to analyze the synchronization mechanism of mouth motion and speech utterance. We also explained a basic principle for realizing smooth interaction through an analysis of Japanese stand-up comedy (manzai).