Collaborative Research: CISE: Large: Executing Natural Instructions in Realistic Uncertain Worlds

机器人要想像人类助手一样流畅地工作，就必须能够接受人类的自然语言指令，并在复杂、不确定的环境中执行这些指令。虽然有商品机器人能够执行各种有用的指令，但目前的人工智能（AI）框架无法完全理解和自主执行大多数指令。相反，当前用于机器人指令遵循的AI技术通常限于高度受限的、不现实的环境和不自然的、脆弱的和刚性的指令格式。该项目的总体目标是研究基本的人工智能原则，使机器人能够在现实的、不确定的世界中可靠地执行自然指令。该项目有可能大幅增加社会可用的体力劳动，而不增加人力劳动量，使典型的人类工人能够指导半自动机器人完成大量日常任务。这只有在机器在自然环境中很容易被人类操纵的情况下才有可能，人类只需要最少的专业训练。考虑到美国需要提高建设实体基础设施的能力，这种设想的劳动力乘数极其相关。例如，这包括新建和升级桥梁和能源系统等公共基础设施，以及高效建造负担得起的住房。这些进步也将对经济的其他部分产生更广泛的影响，如物流，医疗保健，家庭助理。该项目还将通过K-12计划、本科生研究经验和招募代表性不足的研究生人才来促进教育和推广。该项目将设计和开发一个新的嵌入式AI代理集成框架，该框架由计算机视觉、语言理解、世界建模、规划和控制方面的协同进步组成。该框架将通过逐步增加功能的计划进行发展，从逐步执行指令开始，并逐步执行一般类型的面向目标的指令。该研究将使用商品机器人在物理逼真的仿真环境和现实环境中测试和演示该框架。此外，将在每个能力阶段进行用户研究，以便将工作重点放在最终用户效用上。该框架的核心是时空场景的新知识结构，即多模态实体图（MEM），它基于视觉和语言进行更新，并用于规划和技能执行。该研究将研究3D视觉和语言理解方面的新想法，以捕捉环境中的不确定性的方式，根据现实输入不断维护MEM。该项目还将研究一种新的方法来控制机器人技能的低层次全身控制，其灵感来自语言建模的最新成功，这有助于模块化技能学习和知识共享。最后，该项目将通过研究在MEMs上学习动态模型的新想法来提高自动规划能力，该模型被一种基于动态条件语言模型的高级技能规划的新方法所使用。重要的是，所有这些创新将以同步的方式开发，以便对综合框架进行严格的测试和演示。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估来支持。

For robots to fluidly operate as human assistants they must be able to take natural language instructions from humans and act to achieve those instructions in complex, uncertain environments. While there are commodity robots that are physically capable of carrying out a wide variety of useful instructions, current artificial intelligence (AI) frameworks are not able to fully understand and autonomously execute most of those instructions. Rather, current AI techniques for robot instruction following have generally been limited to highly constrained, unrealistic environments and instruction formats that are unnatural, brittle, and rigid. The overarching goal of the project is to study the fundamental AI principles that enable robots to reliably execute natural instructions in realistic, uncertain worlds. The project has the potential to dramatically increase the physical labor available to society, without increasing the amount of human labor, by enabling typical human workers to direct semi-automated robots for a multitude of mundane tasks. This will only be possible if the machines are easily instructable in natural environments by humans who only require minimal specialized training. This envisioned labor multiplier is extremely relevant given the need for increased capacity to build physical infrastructure in the US. This includes, for example, new and upgraded public infrastructure such as bridges and energy systems, as well as efficient construction of affordable housing. These same advances will also result in broader impacts to other parts of the economy, such as logistics, healthcare, household assistants. The project will also contribute to education and outreach through K-12 initiatives, undergraduate research experiences, and recruiting of underrepresented graduate student talent.The project will design and develop a novel integrated framework for embodied AI agents that is comprised of synergistic advances in computer vision, language understanding, world modeling, planning, and control. The framework will evolve over a staged plan of increasing capabilities, starting with step-by-step instruction execution and progressing to executing general types of goal-oriented instructions. The research will test and demonstrate the framework in both physically-realistic simulation environments and real-world environments using commodity robots. In addition, user studies will be conducted at each capability stage to focus the work toward end-user utility. Central to the framework is a new knowledge structure for spatio-temporal scenes, the multi-modal entity map (MEM), which is updated based on vision and language and used for both planning and skill execution. The research will study new ideas in 3D vision and language understanding for continually maintaining the MEM based on realistic inputs in a way that captures uncertainty in the environment. The project will also study a new approach to low-level full-body control for robot skills, inspired by recent successes in language modeling, that facilitates both modular skill learning and knowledge sharing. Finally, the project will advance automated planning capabilities by studying new ideas for learning dynamics models over the MEMs, which are used by a novel approach to high-level skill planning based on dynamics-conditioned language models. Importantly all of these innovations will be developed in a synchronized way to allow for rigorous testing and demonstration of the integrated framework.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.