CAREER: Inferring Minimal but Sufficient Environment Models from Natural Language and Semantic Perception for Collaborative Robots in Dynamic Environments

协作机器人有望对社会产生变革性的影响。在过去的十年中，感知、规划、映射、定位和驱动方面的重大进展加速了机器人从受控实验室环境向复杂和动态人类世界中的各种应用的迁移。随着机器人从独立执行的专业应用过渡到人机团队中的重要角色，有效沟通的能力变得越来越重要。 虽然智能语音助手近年来由于自动语音识别的进步而变得普遍，但这些交互并不以从传感器观察推断的环境模型为条件，也不会导致对其周围环境的物理操纵。最近在用于人机交互的接地语言通信方面取得的进展，其中语音和文本都告知并引导机器人对世界的理解和行动，已经将机器人和人类可以交换的概念的有限空间转变为可以由许多不同对象组成的丰富表示，空间关系，规划约束和动作。 可能存在具有定制环境表示或针对特定应用定制的模型的点解决方案，但是当前的方法无法在非平凡的时间尺度上跨许多场景有效地解释人类指令的含义。 从根本上讲，新的方法接地语言沟通，流畅的原因，过去的动态，目前的配置和/或未来的状态，从选择的观察对象需要协作机器人解释语言交互快速，准确地在人类机器人团队。这个教师早期职业发展（CAREER）项目旨在重新审视协作机器人的算法，模型和智能架构，并重新制定概率图形模型是如何构建和解决的，以便在具有复杂动态，非平凡启示和丰富语义表示的环境中进行有效的接地语言交流。这项研究将通过开发解决两个关键技术差距的算法来研究这个主题。 首先，这项研究将开发算法，推断需要哪些过去和/或现在的观察和/或未来的预测来构建最小但足够的环境模型来解释自然语言。 其次，这项研究将开发算法，迭代更新他们的解决方案的基础上，环境模型的微小修改和增量更新观测和环境预测的符号表示的增强。 这些算法将为自然语言符号基础提供一种新的智能架构，该架构可以推理哪些观察和/或环境投影以及需要从这些动态环境表示中获取哪些信息，以实现可扩展和有能力的模型，用于与协作机器人进行基础语言通信。 这些想法的实验评估将在各种机器人操纵器、现场机器人和移动的操纵器上进行，此外还有专门为教学和推广活动开发的协作移动的机器人。由工程局（ENG）和计算机与信息科学与工程局（CISE）共同管理和资助该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Collaborative robots are poised to have a transformative impact on society. Significant advances in perception, planning, mapping, localization, and actuation over the past decade have accelerated the migration of robots from controlled laboratory environments to diverse applications in a complex and dynamic human world. The ability to effectively communicate has become increasingly important as robots transition from specialized applications performed in isolation to significant roles in human-robot teams. While smart voice assistants have become prevalent in recent years due to advances in automatic speech recognition, those interactions are not conditioned on a model of the environment inferred from sensor observations and do not lead to the physical manipulation of their surroundings. Recent advances in grounded language communication for human-robot interaction, where speech and text both inform and guide a robot's understanding of and actions in the world, have transformed the limited space of concepts that robots and humans could exchange into rich representations that could be composed of many different objects, spatial relationships, planning constraints, and actions. Point solutions with bespoke environment representations or models tailored to specific applications may exist, but no current approach is able to efficiently interpret the meaning of human instructions across many scenarios over non-trivial timescales. Fundamentally new approaches to grounded language communication that fluidly reason about the past dynamics, present configuration and/or future state of objects from select observations are needed for collaborative robots to interpret linguistic interactions quickly and accurately in human-robot teams. This Faculty Early Career Development (CAREER) project seeks to revisit algorithms, models, and intelligence architectures for collaborative robots and reformulate how probabilistic graphical models are constructed and solved for efficient grounded language communication in environments with complex dynamics, non-trivial affordances, and rich semantic representations.This research will investigate this topic by developing algorithms that address two critical technical gaps. First, this research will develop algorithms that infer which past and/or present observations and/or future projections in time are needed to construct minimal but sufficient environment models for interpreting natural language. Second, this research will develop algorithms that iteratively update their solutions based on minor modifications of the environment model and augmentation of symbolic representations from incrementally updating observations and environment projections. These algorithms will enable a novel intelligence architecture for natural language symbol grounding that reasons about which observations and/or environment projections and what kinds of information are needed from these dynamic environment representations to enable scalable and capable models for grounded language communication with collaborative robots. Experimental evaluation of these ideas will be performed on a variety of robot manipulators, field robots, and mobile manipulators in addition to purposefully built collaborative mobile robots that will be developed for teaching and outreach activities.This project is supported by the cross-directorate Foundational Research in Robotics program, jointly managed and funded by the Directorates for Engineering (ENG) and Computer and Information Science and Engineering (CISE).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.