HRI: Perceptually Situated Human-Robot Dialog Models

人类自然会使用对话和手势来讨论复杂的现象和计划，特别是当他们在相互交流时提到环境的物理方面时。现有的机器人视觉系统可以感知人和环境，但它们检测人们经常依赖的详细对话线索(如头部姿势、眼睛凝视和身体姿势)并在多模式对话中利用这些线索的能力有限。计算机视觉的最新进展使追踪如此详细的线索成为可能。机器人可以使用被动测量来感知人们的存在，估计他们的注意力焦点和身体姿势，并识别人类的手势和身体参照。但他们将这些信息集成到自然语言模型中的手段有限；到目前为止，他们使用的是特定领域的对话模型和/或仅限于一对一的交互。另外，自然语言处理的最新进展导致了对话模型的出现，该模型可以跟踪多个参与者之间相对自由的对话，并提取关于人们意图和行为的有意义的语义。这些多方对话模型已用于会议环境和其他领域。在这个项目中，PI和他的团队将融合这两条研究路线，以实现一种感知上的、自然的对话模型，机器人可以使用该模型与人进行多模式交互。他们将开发一种合理的通用对话模型，允许处于环境中的代理跟踪它周围的对话，知道它何时被寻址，并从人类操作员那里获得指示，关于它应该在哪里找到或放置各种对象，它应该在环境中寻找什么，以及它应该关注、跟踪或服从哪些人。项目成果将把现有的对话管理技术扩展到更一般的交互管理理论，还将扩展当前最先进的视觉研究，使其能够识别非语言对话提示的微妙之处，以及将这些提示与正在进行的对话解释和与世界的互动相结合的方法。广泛的影响：显然，这项研究将产生许多积极的社会影响。最终，开发有效的人-机器人接口将允许更多地部署机器人来执行否则人类必须执行的危险任务，并将使更多地使用机器人在家庭环境中执行服务任务。作为该项目的一部分，国际和平研究所将开展外联工作，吸引中学生参与，希望接触到人力资源研究所的研究可以提高他们对科学和工程研究的兴趣。

Humans naturally use dialog and gestures to discuss complex phenomena and plans, especially when they refer to physical aspects of the environment while they communicate with each other. Existing robot vision systems can sense people and the environment, but are limited in their ability to detect the detailed conversational cues people often rely upon (such as head pose, eye gaze, and body gestures), and to exploit those cues in multimodal conversational dialog. Recent advances in computer vision have made it possible to track such detailed cues. Robots can use passive measures to sense the presence of people, estimate their focus of attention and body pose, and to recognize human gestures and identify physical references. But they have had limited means of integrating such information into models of natural language; heretofore, they have used dialog models for specific domains and/or were limited to one-on-one interaction. Separately, recent advances in natural language processing have led to dialog models that can track relatively free-form conversation among multiple participants, and extract meaningful semantics about people's intentions and actions. These multi-party dialog models have been used in meeting environments and other domains. In this project, the PI and his team will fuse these two lines of research to achieve a perceptually situated, natural conversation model that robots can use to interact multimodally with people. They will develop a reasonably generic dialog model that allows a situated agent to track the dialog around it, know when it is being addressed, and take direction from a human operator regarding where it should find or place various objects, what it should look for in the environment, and which individuals it should attend to, follow, or obey. Project outcomes will extend existing dialog management techniques to a more general theory of interaction management, and will also extend current state-of-the-art vision research to be able to recognize the subtleties of nonverbal conversational cues, as well as methods for integrating those cues with ongoing dialog interpretation and interaction with the world.Broader Impacts: There are clearly many positive societal impacts that will derive from this research. Ultimately, development of effective human-robot interfaces will allow greater deployment of robots to perform dangerous tasks that humans would otherwise have to perform, and will also enable greater use of robots for service tasks in domestic environments. As part of the project, the PI will conduct outreach efforts to engage secondary-school students in the hope that exposure to HRI research may increase their interest in science and engineering studies.