II-NEW: Towards an Infrastructure for Research on Multimodal Language Processing in Situated Human Robot Dialogue

新一代机器人正在涌现，旨在每天与人类互动，提供服务、护理和陪伴。为了支持人与这类机器人之间的自然交互，支持情景人类机器人对话的健壮语言处理将变得越来越重要。与传统的口语对话系统和多通道对话界面相比，情景人类机器人对话由于两个独特的特点而截然不同。第一个特点是情境性。机器人位于一个与人类伴侣共同居住的物理世界中。机器人、人和环境之间的空间关系，以及环境的动态性质，对机器人如何完成任务和与人类交互有着巨大的影响。第二个特征是具体化。机器人和它的人类伴侣在环境中都有实体。在体验式交际中，说话人广泛使用非语言形式(例如，眼睛凝视和手势)来进行对话，并提及共享的环境。这两个特点使得情景人类机器人对话中人类语言的自动解释极具挑战性。这笔资金将为PI及其团队提供增强的基础设施，使他们能够应对这些挑战。新资源将包括物理环境和虚拟环境，以实现以人为中心的调查；捕获人类多模式语言行为的工具，包括在情景中的人类机器人对话期间的语音、眼睛凝视和手势；以及支持绿野仙踪实验、数据收集和数据分析的系统。物理世界和虚拟世界的融合是新基础设施的创新。传感器和效应器技术的局限性往往使改变机器人配置和实现所需行为变得困难或昂贵。虚拟世界范式允许高效和高保真地模拟物理世界和机器人，以及研究在许多不同条件下的多模式语言行为，否则在物理世界中很难获得这些行为。新的基础设施将通过促进各种受控实验，实现以人为中心的方法。它将使新的经验发现成为可能，并为心理语言学上可信的多通道语言处理高级技术提供一个试验台。广泛的影响：新的基础设施将为PI和她在密歇根州立大学的同事提供巨大的研究和合作机会。它将对实现下一代社交和认知机器人产生深远的影响。这一基础设施还将通过研究、指导和课程开发为密歇根州立大学的学生提供新的令人兴奋的培训和教育机会。它将通过密歇根州立大学的几个外展项目，为K-12学生带来新的教育体验，并鼓励他们更广泛地参与工程。

A new generation of robots is emerging which aims to interact with humans on a daily basis to provide service, care, and companionship. To support natural interaction between people and this type of robot, robust language processing enabling situated human robot dialogue will become increasingly important. Compared to traditional spoken dialogue systems and multimodal conversational interfaces, situated human robot dialogue is drastically different due to two unique characteristics. The first characteristic is situatedness. A robot is situated in a physical world that is cohabited with human partners. The spatial relations between the robot, the human, and the environment, and the dynamic nature of the surroundings, have a massive influence on how the robot accomplishes its task and interacts with the human. The second characteristic is embodiment. A robot and its human partner both have physical bodies in the environment. In embodied communication, speakers make extensive use of non-verbal modalities (e.g., eye gaze and gestures) to engage in conversation and make reference to the shared environment. These two characteristics make automated interpretation of human language in situated human robot dialogue extremely challenging. This is funding to provide the PI and her team with an enhanced infrastructure that will enable them to address these challenges. The new resources will include a physical environment and a virtual environment to enable human-centered investigation, tools to capture human multimodal language behaviors that include human speech, eye gaze, and gesture during situated human robot dialogue, and systems to support Wizard-of-Oz experiments, data collection, and data analysis. The integration of a physical world and a virtual world is an innovation of the new infrastructure. Limitations of sensor and effector technology often make it difficult or expensive to change robot configurations and implement desired behaviors. The virtual world paradigm allows efficient and high fidelity simulation of the physical world and robots, as well as studies on multimodal language behavior under many different conditions which are otherwise difficult to obtain in the physical world. The new infrastructure will enable a human-centered approach by facilitating a wide variety of controlled experiments. It will enable new empirical findings and provide a testbed for advanced techniques for multimodal language processing that are psycholinguistically plausible.Broader Impacts: The new infrastructure will provide tremendous research and collaborative opportunities for the PI and her colleagues at Michigan State University. It will have profound implications in enabling the next generation of social and cognitive robots. This infrastructure will also provide new and exciting training and education opportunities for students at MSU through research mentoring and curriculum development. It will bring new educational experiences to K-12 students and encourage broader participation in engineering through several outreach programs at MSU.