NRI: Collaborative Research: Jointly Learning Language and Affordances

这个项目的研究人员设想了一个机器人包围我们的世界，在我们的家里，在我们的医院，在我们的工厂里，通过运送药物、准备食物和组装物品来帮助人们。实现这一愿景需要机器人与人们就他们的需求进行沟通，然后计划他们的活动以帮助满足这些需求。以前的研究分别解决了这两个问题，导致技术解决方案在现实世界中不可靠地工作，并导致人与机器人通信的困难。为了解决这些问题，我们正在开发带有Affordance的物理接地语言(PGLA)框架，并将我们的研究集中在两个方面：1)使机器人能够观察患者，然后回答护士关于患者活动的问题；2)使机器人能够在协作烹饪任务和制造环境中响应自然语言要求。我们将公布我们的开源数据集和代码，这将对机器人以外的其他技术领域产生影响，如计算机视觉和机器学习。我们提出的研究结果将直接应用于制造和辅助机器人等行业。该项目采用概率方法来共同学习识别环境中的负担并预测相关的自然语言请求和描述。由于启示地图以感知数据为基础，我们的机器人将学会强健地操纵物理世界中的物体，对自然语言命令做出反应，并用语言描述它们的经历。我们的学习方法使机器人能够从进行活动的人的大型数据集中推断跨模型知识，并与活动的自然语言描述配对，利用每种通道的优势通知其他通道。我们的新学习算法将集成和学习多领域数据库，如语义网、视觉场景，以及与自然语言描述配对的新活动数据库。

The investigators of this project envision a world where robots surround us, in our homes, in our hospitals, and in our factories, helping people by delivering medicine, preparing food, and assembling objects. Achieving this vision requires robots to communicate with people about their needs, and then plan their activities to help meet those needs. Previous research has addressed these two problems separately, leading to technical solutions that do not work reliably in real-world situations, and to difficulties in human-robot communication. To solve these problems, we are developing the Physically-Grounded Language with Affordances (PGLA) framework and concentrate our research into two thrusts: 1) enable a robot to observe a patient, then answer a nurse's questions about the patient's activity, and 2) enable a robot to respond to natural language requests in a collaborative cooking task and in a manufacturing setting. We will release our open-source data sets and code, which will have impact in other technical areas beyond robotics, such as computer vision and machine learning. The results of our proposed research will find direct applications in industries such as manufacturing and assistive robotics.This project takes a probabilistic approach to jointly learn to recognize affordances in the environment and predict associated natural language requests and descriptions. Since the affordance map is grounded to perceptual data, our robots will learn to robustly manipulate objects in the physical world, respond to natural language commands, and describe their experiences using words. Our learning approach enables the robot to infer cross-model knowledge from large data sets of people carrying out activities paired with natural language descriptions of the activities, leveraging the strength of each modality to inform the others. Our novel learning algorithms will integrate and learn from multi-domain databases such as the semantic web, visual scenes, and a novel activity database paired with natural language descriptions.