Building joint models of language and the 3D world

我的研究旨在将自然语言与世界的3D表示联系起来。人们生活在一个物理的、三维的世界里。我们使用自然语言来谈论世界以及我们与世界的互动。物理世界的视觉表现和自然语言是人类交流的两种重要的、相互联系的方式。为了创造能够理解和帮助我们日常生活的人工智能，我们需要赋予机器谈论和理解物理世界的能力。我的目标是开发算法和模型，使人工智能助手(或机器人)能够将参考信息(如餐桌上的黑色马克杯)解析为物体的3D表示。此外，人工智能还应该理解与命令相关的物理世界中的状态变化，比如“把杯子从桌子上拿到床左边的床头柜”。我们如何开发能够与人们谈论真实世界的人工智能？有几个研究问题需要解决：理解人们在3D环境中指的是什么，如何与世界交互，以及人们使用什么语言来描述交互，以及解析涉及多个代理的世界中的复杂对话。我的研究解决了这些问题，并探索了一些基本问题，这些问题支撑着对语言和视觉世界之间联系的人类层面的理解。在这一具有挑战性的问题领域取得的进展可以产生巨大的影响。通过解决这个问题，我们可以使机器人助手能够使用自然语言与我们交流关于我们生活的世界。我们还可以通过语言驱动的用户界面更轻松地创建2D和3D内容。另一个应用是在教育环境中，我们可以使用学习到的语言和视觉概念之间的映射来向虚拟环境中的外语使用者传授这些概念是如何在一种新语言中刻画出来的。例如，想象一下一个虚拟现实游戏，你可以透过起居室看到用西班牙语描述的所有物体。除了这些具体的应用，在共同理解语言和物理世界方面的进展将构成一般人工智能的基础，也是帮助将机器感知和符号推理结合在一起的因素之一，这是人工智能中一个长期存在的重大挑战问题。这一领域的进展将加强加拿大学术机构在人工智能和机器学习相关主题上的实力和全球领先地位。这一建议旨在通过以下贡献来构建语言与世界的联合模型：1)建立一个在线交互模拟平台，在该平台中，多人对话可以同时进行3D环境的操纵。2)研究联合模拟世界和语言的表征。3)利用这些表示法来设计能够将语言与物理世界联系起来的算法。

My research aims to connect natural language with 3D representations of the world. People live in a physical, three-dimensional world. We use natural language to talk about the world and our interactions with the world. Visual representations of the physical world and natural language are two critical, interconnected modalities for human communication. To create AI that can understand and assist us in our daily lives, we need to give machines the ability to talk about and understand the physical world. My aim is to develop algorithms and models that can allow an AI assistant (or robot) to resolve references such as "black mug on the dining table" to the 3D representation of the object. Moreover, the AI should understand the state change in the physical world associated with commands such as "take the mug from the table to the nightstand on the left of the bed". How do we develop AI that can talk with people about the real world? There are several research problems to address: understanding what people are referring to in a 3D environment, how to interact with the world and what language people use to describe interactions, and parsing complex dialogue in the world involving multiple agents. My research addresses these problems and explores fundamental questions that underpin a human-level understanding of the connection between language and the visual world. Progress in this challenging problem area can have immense impact. By tackling this problem, we can enable robotic assistants that can communicate with us using natural language about the world we live in. We can also make 2D and 3D content creation easier through language-driven UIs. Another application is in educational settings where we can use the learned mapping between language and visual concepts to teach foreign language speakers in virtual environments about how these concepts are carved out in a new language. Imagine, for example, a virtual reality game where you can look through your living room and see all the objects described in Spanish. Beyond these concrete applications, progress towards jointly understanding language and the physical world will form a foundation for general artificial intelligence, and is one ingredient for helping to combine machine perception and symbolic reasoning, a long-standing grand challenge problem in AI. Progress in this area will reinforce the strength and global leadership of Canadian academic institutions on topics related to artificial intelligence and machine learning. This proposal aims to build a joint model of language and the world by making the following contributions: 1) Build an online interactive simulation platform in which multiplayer dialogue can happen alongside manipulation of a 3D environment. 2) Investigate representations that jointly model the world and language. 3) Leverage these representations to design algorithms that can link language to the physical world.