NRI: Collaborative Research: Jointly Learning Language and Affordances

该项目的研究人员设想了一个机器人围绕在我们身边的世界，在我们的家里，在我们的医院，在我们的工厂，通过运送药品，准备食物和组装物品来帮助人们。要实现这一愿景，机器人需要与人沟通他们的需求，然后计划他们的活动来帮助满足这些需求。之前的研究分别解决了这两个问题，导致技术解决方案在现实世界中不可靠地工作，并且在人机通信中遇到困难。为了解决这些问题，我们正在开发具有能力的物理基础语言（PGLA）框架，并将我们的研究集中在两个重点上：1)使机器人能够观察病人，然后回答护士关于病人活动的问题；2)使机器人能够在协作烹饪任务和制造环境中响应自然语言请求。我们将发布我们的开源数据集和代码，这将对机器人以外的其他技术领域产生影响，例如计算机视觉和机器学习。我们提出的研究结果将直接应用于制造业和辅助机器人等行业。该项目采用概率方法，共同学习识别环境中的启示，并预测相关的自然语言请求和描述。由于功能图是基于感知数据的，我们的机器人将学会在物理世界中强大地操纵物体，对自然语言命令做出反应，并使用文字描述他们的经历。我们的学习方法使机器人能够从进行活动的人的大型数据集中推断跨模型知识，并与活动的自然语言描述配对，利用每种模态的强度来通知其他模态。我们的新学习算法将集成和学习多领域数据库，如语义网、视觉场景和与自然语言描述配对的新活动数据库。

The investigators of this project envision a world where robots surround us, in our homes, in our hospitals, and in our factories, helping people by delivering medicine, preparing food, and assembling objects. Achieving this vision requires robots to communicate with people about their needs, and then plan their activities to help meet those needs. Previous research has addressed these two problems separately, leading to technical solutions that do not work reliably in real-world situations, and to difficulties in human-robot communication. To solve these problems, we are developing the Physically-Grounded Language with Affordances (PGLA) framework and concentrate our research into two thrusts: 1) enable a robot to observe a patient, then answer a nurse's questions about the patient's activity, and 2) enable a robot to respond to natural language requests in a collaborative cooking task and in a manufacturing setting. We will release our open-source data sets and code, which will have impact in other technical areas beyond robotics, such as computer vision and machine learning. The results of our proposed research will find direct applications in industries such as manufacturing and assistive robotics.This project takes a probabilistic approach to jointly learn to recognize affordances in the environment and predict associated natural language requests and descriptions. Since the affordance map is grounded to perceptual data, our robots will learn to robustly manipulate objects in the physical world, respond to natural language commands, and describe their experiences using words. Our learning approach enables the robot to infer cross-model knowledge from large data sets of people carrying out activities paired with natural language descriptions of the activities, leveraging the strength of each modality to inform the others. Our novel learning algorithms will integrate and learn from multi-domain databases such as the semantic web, visual scenes, and a novel activity database paired with natural language descriptions.