ITR: Multimodal Human Computer Interaction: Toward a Proactive Computer

这是一项为期五年的持续奖励的第一年资助。这个项目是基于这样一种信念：为了让普通大众更容易接触到计算机，计算机必须在与人的互动中更主动。在人与人之间的互动中，如果一个人在做出任何沟通尝试之前都要等待每一个命令，那么他就会被认为是不合作和无益的。为了使计算机更主动，从而承担起它在交互中发起的那部分负担，它必须(1)拥有更多关于其用户的实时信息，(2)基于这些信息而不是简单地根据用户命令选择操作的算法。计算机需要用户当前和过去的情感、动机和认知状态以及手头任务的状态等信息。需要一个理论来指导算法的开发，根据用户和任务状态选择适当的操作。这项研究构成了PI的多学科团队开发这种能力的下一步尝试。建议的研究包括：(1)进一步开发感知用户姿势、动作、表情和语音的方法；(2)对这些信息进行分析和融合，识别和跟踪用户状态；(3)任务状态跟踪；(4)创建一个情感和动作标记的录像带语料库，用于计算机学习；(5)情感沟通的进一步发展；(6)发展以人为本的基于国家的行动决策基础；(7)评价计算机对人类行为和反应的作用。测试平台是一个实践教育的环境，在科学和工程，使用乐高Mindstorms建筑和机器人环境，与中学年龄的孩子。重点将放在开发积极主动的计算方法，以鼓励少数民族儿童和妇女的兴趣和概念发展，他们在科学上往往表现出较低的成就。虽然这项工作将在教育环境中进行，但所开发和研究的方法将广泛适用，本项目应作为在其他计算机辅助情况下所需的工作类型的范例。

This is the first year funding of a five-year continuing award. This project is based on the belief that, to be more accessible to the general population, computers must be more proactive in their interactions with people. In human interaction, someone who waits for each command before making any communication attempt would be regarded as uncooperative and unhelpful. In order for a computer to be more proactive and, thus, to bear its part of the burden of initiation in interactions, it must have (1) much more real-time information about its user, and (2) algorithms that select actions based on this information rather than simply on user commands. The computer needs information about the user's current and past emotional, motivational and cognitive state as well as the state of the task at hand. A theory, is needed to guide the development of algorithms that select appropriate actions based on user and task state. This research constitutes the next steps in an attempt by the PI's multidisciplinary team to develop this capability.Proposed research includes: (1) further development of methods to sense user postures, movements, expressions and speech; (2) analysis and fusion of this information to identify and track user states; (3) task state tracking; (4) creating a corpus of emotion- and action-labeled videotapes for use with computer learning; (5) further development of affective communication; (6) development of the basis for human-centered state-based action decisions; and (7) evaluation of computer proaction on human behavior and response. The testbed is an environment for hands-on education in science and engineering, using the Lego Mindstorms construction and robotics environment, with children of middle school age. An emphasis will be on developing proactive computing methods for encouraging interest and conceptual development of minority children and females, who often show lower achievement in science. Although the work will be conducted within an educational environment, the methods developed and studied will be broadly applicable, and this project should serve as an exemplar of the type of work that is needed in other computer-aided situations.