Building joint models of language and the 3D world

构建语言和 3D 世界的联合模型

• 批准号: RGPIN-2020-07196
• 负责人: Chang, Angel
• 金额: $ 2.91万
• 依托单位: Simon Fraser University
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2021
• 资助国家: 加拿大
• 起止时间: 2021-01-01 至 2022-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=741654
• 关键词: Building; joint; models; language; 3D

My research aims to connect natural language with 3D representations of the world. People live in a physical, three-dimensional world. We use natural language to talk about the world and our interactions with the world. Visual representations of the physical world and natural language are two critical, interconnected modalities for human communication. To create AI that can understand and assist us in our daily lives, we need to give machines the ability to talk about and understand the physical world. My aim is to develop algorithms and models that can allow an AI assistant (or robot) to resolve references such as "black mug on the dining table" to the 3D representation of the object. Moreover, the AI should understand the state change in the physical world associated with commands such as "take the mug from the table to the nightstand on the left of the bed". How do we develop AI that can talk with people about the real world? There are several research problems to address: understanding what people are referring to in a 3D environment, how to interact with the world and what language people use to describe interactions, and parsing complex dialogue in the world involving multiple agents. My research addresses these problems and explores fundamental questions that underpin a human-level understanding of the connection between language and the visual world. Progress in this challenging problem area can have immense impact. By tackling this problem, we can enable robotic assistants that can communicate with us using natural language about the world we live in. We can also make 2D and 3D content creation easier through language-driven UIs. Another application is in educational settings where we can use the learned mapping between language and visual concepts to teach foreign language speakers in virtual environments about how these concepts are carved out in a new language. Imagine, for example, a virtual reality game where you can look through your living room and see all the objects described in Spanish. Beyond these concrete applications, progress towards jointly understanding language and the physical world will form a foundation for general artificial intelligence, and is one ingredient for helping to combine machine perception and symbolic reasoning, a long-standing grand challenge problem in AI. Progress in this area will reinforce the strength and global leadership of Canadian academic institutions on topics related to artificial intelligence and machine learning. This proposal aims to build a joint model of language and the world by making the following contributions: 1) Build an online interactive simulation platform in which multiplayer dialogue can happen alongside manipulation of a 3D environment. 2) Investigate representations that jointly model the world and language. 3) Leverage these representations to design algorithms that can link language to the physical world.

我的研究旨在将自然语言与世界的3D表示联系起来。人们生活在一个物理的、三维的世界里。我们使用自然语言来谈论世界以及我们与世界的互动。物理世界的视觉表示和自然语言是人类交流的两种重要的、相互关联的方式。为了创造能够理解和帮助我们日常生活的人工智能，我们需要让机器能够谈论和理解物理世界。我的目标是开发算法和模型，允许AI助手（或机器人）将引用（如“餐桌上的黑色马克杯”）解析为对象的3D表示。此外，人工智能应该理解与命令相关的物理世界中的状态变化，例如“将杯子从桌子上拿到床左边的床头柜上”。我们如何开发能够与人们谈论真实的世界的人工智能？有几个研究问题需要解决：理解人们在3D环境中所指的是什么，如何与世界互动，人们使用什么语言来描述互动，以及解析世界中涉及多个代理的复杂对话。我的研究解决了这些问题，并探索了支撑人类对语言和视觉世界之间联系的理解的基本问题。在这一具有挑战性的问题领域取得进展可能产生巨大影响。通过解决这个问题，我们可以使机器人助手能够使用自然语言与我们交流我们生活的世界。我们还可以通过语言驱动的UI使2D和3D内容创建更容易。另一个应用是在教育环境中，我们可以使用语言和视觉概念之间的学习映射，在虚拟环境中教授外语使用者如何在新语言中划分这些概念。想象一下，例如，一个虚拟现实游戏，你可以透过你的客厅，看到所有用西班牙语描述的物体。除了这些具体的应用之外，共同理解语言和物理世界的进展将为通用人工智能奠定基础，并且是帮助将联合收割机机器感知和符号推理结合起来的一个要素，这是人工智能长期面临的重大挑战问题。这一领域的进展将加强加拿大学术机构在人工智能和机器学习相关主题方面的实力和全球领导地位。该提案旨在通过以下贡献建立语言和世界的联合模型：1）建立一个在线交互式仿真平台，其中多人对话可以与3D环境的操作一起发生。2)研究共同模拟世界和语言的表示。3)利用这些表示来设计可以将语言与物理世界联系起来的算法。