NRI: Collaborative Research: Jointly Learning Language and Affordances

该项目的研究人员设想了一个机器人围绕着我们的世界，在我们的家中，在我们的医院和工厂，通过提供药物，准备食物和组装物品来帮助人们。实现这一愿景需要机器人与人们沟通他们的需求，然后计划他们的活动以帮助满足这些需求。 以前的研究分别解决了这两个问题，导致技术解决方案在现实世界中无法可靠地工作，并且在人类与机器人通信中存在困难。 为了解决这些问题，我们正在开发具有可供性的物理接地语言（PGLA）框架，并将我们的研究集中在两个方面：1）使机器人能够观察患者，然后回答护士关于患者活动的问题，以及2）使机器人能够在协作烹饪任务和制造环境中响应自然语言请求。我们将发布我们的开源数据集和代码，这将对机器人技术以外的其他技术领域产生影响，例如计算机视觉和机器学习。 我们提出的研究结果将直接应用于制造业和辅助机器人等行业。该项目采用概率方法，共同学习识别环境中的启示，并预测相关的自然语言请求和描述。 由于示能图基于感知数据，我们的机器人将学会稳健地操纵物理世界中的物体，响应自然语言命令，并使用文字描述他们的体验。 我们的学习方法使机器人能够从进行活动的人的大型数据集中推断出跨模型知识，并与活动的自然语言描述配对，利用每种模态的优势来通知其他模态。我们的新型学习算法将整合和学习多领域数据库，如语义网，视觉场景，以及与自然语言描述配对的新型活动数据库。

The investigators of this project envision a world where robots surround us, in our homes, in our hospitals, and in our factories, helping people by delivering medicine, preparing food, and assembling objects. Achieving this vision requires robots to communicate with people about their needs, and then plan their activities to help meet those needs. Previous research has addressed these two problems separately, leading to technical solutions that do not work reliably in real-world situations, and to difficulties in human-robot communication. To solve these problems, we are developing the Physically-Grounded Language with Affordances (PGLA) framework and concentrate our research into two thrusts: 1) enable a robot to observe a patient, then answer a nurse's questions about the patient's activity, and 2) enable a robot to respond to natural language requests in a collaborative cooking task and in a manufacturing setting. We will release our open-source data sets and code, which will have impact in other technical areas beyond robotics, such as computer vision and machine learning. The results of our proposed research will find direct applications in industries such as manufacturing and assistive robotics.This project takes a probabilistic approach to jointly learn to recognize affordances in the environment and predict associated natural language requests and descriptions. Since the affordance map is grounded to perceptual data, our robots will learn to robustly manipulate objects in the physical world, respond to natural language commands, and describe their experiences using words. Our learning approach enables the robot to infer cross-model knowledge from large data sets of people carrying out activities paired with natural language descriptions of the activities, leveraging the strength of each modality to inform the others. Our novel learning algorithms will integrate and learn from multi-domain databases such as the semantic web, visual scenes, and a novel activity database paired with natural language descriptions.