II-NEW: Towards an Infrastructure for Research on Multimodal Language Processing in Situated Human Robot Dialogue

新一代机器人正在出现，其目标是每天与人类互动，提供服务，护理和陪伴。 为了支持人与这种类型的机器人之间的自然交互，支持情景化人类机器人对话的强大语言处理将变得越来越重要。 与传统的口语对话系统和多模式对话界面相比，由于两个独特的特征，情景化的人类机器人对话是截然不同的。 第一个特征是情境性。 机器人位于与人类伴侣同居的物理世界中。 机器人、人类和环境之间的空间关系，以及周围环境的动态特性，对机器人如何完成任务以及与人类的互动有着巨大的影响。 第二个特点是体现。 机器人和它的人类伙伴在环境中都有身体。 在具身交流中，说话者广泛使用非语言形式（例如，眼睛注视和手势）来参与对话并参考共享环境。 这两个特点使得在情景化的人类机器人对话中自动解释人类语言极具挑战性。 这笔资金将为PI及其团队提供增强的基础设施，使他们能够应对这些挑战。 新的资源将包括一个物理环境和一个虚拟环境，以实现以人为中心的调查，捕捉人类多模态语言行为的工具，包括人类语音，眼睛凝视和手势，以及支持绿野仙踪实验，数据收集和数据分析的系统。 物理世界和虚拟世界的融合是新基础设施的创新。 传感器和效应器技术的限制通常使得改变机器人配置和实现期望的行为变得困难或昂贵。 虚拟世界的范例允许高效和高保真的模拟物理世界和机器人，以及在许多不同的条件下，否则很难获得在物理世界中的多模态语言行为的研究。 新的基础设施将通过促进各种受控实验来实现以人为本的方法。 它将使新的实证研究结果，并提供了一个测试平台，为先进的技术，多模态语言处理是心理语言学上plausible.Broader影响：新的基础设施将提供巨大的研究和合作机会，为PI和她的同事在密歇根州立大学。 它将对下一代社交和认知机器人产生深远的影响。 该基础设施还将通过研究指导和课程开发为MSU的学生提供新的和令人兴奋的培训和教育机会。 它将为K-12学生带来新的教育体验，并通过密歇根州立大学的几个外展计划鼓励更广泛地参与工程。

A new generation of robots is emerging which aims to interact with humans on a daily basis to provide service, care, and companionship. To support natural interaction between people and this type of robot, robust language processing enabling situated human robot dialogue will become increasingly important. Compared to traditional spoken dialogue systems and multimodal conversational interfaces, situated human robot dialogue is drastically different due to two unique characteristics. The first characteristic is situatedness. A robot is situated in a physical world that is cohabited with human partners. The spatial relations between the robot, the human, and the environment, and the dynamic nature of the surroundings, have a massive influence on how the robot accomplishes its task and interacts with the human. The second characteristic is embodiment. A robot and its human partner both have physical bodies in the environment. In embodied communication, speakers make extensive use of non-verbal modalities (e.g., eye gaze and gestures) to engage in conversation and make reference to the shared environment. These two characteristics make automated interpretation of human language in situated human robot dialogue extremely challenging. This is funding to provide the PI and her team with an enhanced infrastructure that will enable them to address these challenges. The new resources will include a physical environment and a virtual environment to enable human-centered investigation, tools to capture human multimodal language behaviors that include human speech, eye gaze, and gesture during situated human robot dialogue, and systems to support Wizard-of-Oz experiments, data collection, and data analysis. The integration of a physical world and a virtual world is an innovation of the new infrastructure. Limitations of sensor and effector technology often make it difficult or expensive to change robot configurations and implement desired behaviors. The virtual world paradigm allows efficient and high fidelity simulation of the physical world and robots, as well as studies on multimodal language behavior under many different conditions which are otherwise difficult to obtain in the physical world. The new infrastructure will enable a human-centered approach by facilitating a wide variety of controlled experiments. It will enable new empirical findings and provide a testbed for advanced techniques for multimodal language processing that are psycholinguistically plausible.Broader Impacts: The new infrastructure will provide tremendous research and collaborative opportunities for the PI and her colleagues at Michigan State University. It will have profound implications in enabling the next generation of social and cognitive robots. This infrastructure will also provide new and exciting training and education opportunities for students at MSU through research mentoring and curriculum development. It will bring new educational experiences to K-12 students and encourage broader participation in engineering through several outreach programs at MSU.