Understanding human intention and activities for versatile real-time human-machine-interactions

在21世纪的世纪，我们的个人和社会活动是在两个不同的领域进行的：物理真实的世界和网络社会。为了实现这些领域的平滑和随意的集成，我们需要一个新的交互模型，它比现有的“命令-响应”模型更进一步。在本研究中，我们提出了“人机共生系统”的概念作为下一代交互模式。人机共生系统的特点是这样一个特征，即系统为人类工作，即使它们没有明确的顺序。为了实现这种共生系统，需要设计实时的双边多模态信息处理机制：通过观察人类行为，了解人类的活动和意图，并在适当的时机提供适当的信息，自愿支持人类的行为。为了实现人机共生系统，我们进行了以下基础性研究，取得了包括理论模型和真实的世界相互作用系统在内的一系列重要成果：1.观察人类活动（识别系统）：我们开发了一个摄像机网络系统和可穿戴主动视觉传感器系统，用于基于实时3D运动测量来理解人类的意图和活动。吸引人（生成系统）：我们开发了一个智能桌面系统，它根据双手操作桌面板上的投影对象，在桌面板上呈现多功能信息。与人交互（交互模型）：我们提出了一个动态事件表示模型，我们将其称为“混合动态系统”。通过将该模型应用于细粒度人脸表情分类，并分析嘴部运动和语音发声的同步机制，验证了该模型的有效性。通过对日本单口相声的分析，说明了实现流畅互动的基本原则。

In the 21st century, our personal and social activities are conducted in two different domains: physical real world and cyber network society. For realizing smooth and casual integration of these domains, we need a novel interaction model that goes a step further beyond the existing "command-and-response model."In this research, we proposed a concept of "man-machine symbiotic systems" as the next generation interaction model. Man-machine symbiotic systems are characterized by such a feature that the systems work for humans even if they are not explicitly ordered. For realizing such a symbiotic system, it is required to design real-time bi-lateral multimodal information processing mechanisms: observing human behaviors to understand their activities and intentions, and supporting them voluntarily by presenting appropriate information at an appropriate timing.To realize man-machine symbiotic systems, we conducted the following fundamental researches and obtained a number of important results including theoretical models and real world interaction systems:1. Observing Human Activities (recognition system): We developed a camera-network system and wearable active vision sensor systems for understanding human intentions and activities based on realtime 3D motion measurement.2. Attracting Human (generation system): We developed an intelligent desk system, which presents versatile information on a desk top plate according to two-handed manipulations of projected objects on the plate.3. Interacting with Human (interaction model): We proposed a dynamic event representation model which we refer to as a "hybrid dynamical system". We verified the effectiveness of the model by applying it to classify fine-grained facial expression, and to analyze the synchronization mechanism of mouth motion and speech utterance. We also explained a basic principle for realizing smooth interaction through an analysis of Japanese stand-up comedy (manzai).