ITR: Multimodal Human Computer Interaction: Toward a Proactive Computer

这是一个为期五年的连续奖励的第一年资助。 这个项目是基于这样一种信念，即为了更好地为大众所用，计算机必须在与人的互动中更加积极主动。 在人际交往中，等待每个命令才进行任何交流尝试的人会被认为是不合作和不乐于助人的。 为了让计算机更主动，从而承担起交互中启动的部分负担，它必须（1）拥有更多关于用户的实时信息，以及（2）基于这些信息而不是简单地根据用户命令选择操作的算法。 计算机需要有关用户当前和过去的情感、动机和认知状态以及手头任务状态的信息。 需要一种理论来指导算法的开发，该算法根据用户和任务状态选择适当的动作。 这项研究构成了PI多学科团队开发这一能力的下一步尝试。拟议的研究包括： (1)进一步开发感知用户姿势、动作、表情和语音的方法; (2)分析和融合这些信息，以识别和跟踪用户状态; (3)任务状态跟踪; (4)创建一个情感和动作标记的录像带语料库，用于计算机学习; (5)进一步发展情感交流;（6）发展以人为本的基于状态的行动决策的基础; （7）评价计算机对人的行为和反应的主动性。 该试验台是一个环境，在科学和工程动手教育，使用乐高Mindstorms建设和机器人环境，与中学生年龄的儿童。 重点将是发展积极主动的计算方法，以鼓励少数民族儿童和妇女的兴趣和概念发展，他们往往在科学方面表现出较低的成就。 虽然这项工作将在教育环境中进行，但开发和研究的方法将广泛适用，该项目应作为其他计算机辅助情况下所需的工作类型的典范。

This is the first year funding of a five-year continuing award. This project is based on the belief that, to be more accessible to the general population, computers must be more proactive in their interactions with people. In human interaction, someone who waits for each command before making any communication attempt would be regarded as uncooperative and unhelpful. In order for a computer to be more proactive and, thus, to bear its part of the burden of initiation in interactions, it must have (1) much more real-time information about its user, and (2) algorithms that select actions based on this information rather than simply on user commands. The computer needs information about the user's current and past emotional, motivational and cognitive state as well as the state of the task at hand. A theory, is needed to guide the development of algorithms that select appropriate actions based on user and task state. This research constitutes the next steps in an attempt by the PI's multidisciplinary team to develop this capability.Proposed research includes: (1) further development of methods to sense user postures, movements, expressions and speech; (2) analysis and fusion of this information to identify and track user states; (3) task state tracking; (4) creating a corpus of emotion- and action-labeled videotapes for use with computer learning; (5) further development of affective communication; (6) development of the basis for human-centered state-based action decisions; and (7) evaluation of computer proaction on human behavior and response. The testbed is an environment for hands-on education in science and engineering, using the Lego Mindstorms construction and robotics environment, with children of middle school age. An emphasis will be on developing proactive computing methods for encouraging interest and conceptual development of minority children and females, who often show lower achievement in science. Although the work will be conducted within an educational environment, the methods developed and studied will be broadly applicable, and this project should serve as an exemplar of the type of work that is needed in other computer-aided situations.