Building joint models of language and the 3D world

点击翻译按钮获取中文摘要

My research aims to connect natural language with 3D representations of the world. People live in a physical, three-dimensional world. We use natural language to talk about the world and our interactions with the world. Visual representations of the physical world and natural language are two critical, interconnected modalities for human communication. To create AI that can understand and assist us in our daily lives, we need to give machines the ability to talk about and understand the physical world. My aim is to develop algorithms and models that can allow an AI assistant (or robot) to resolve references such as "black mug on the dining table" to the 3D representation of the object. Moreover, the AI should understand the state change in the physical world associated with commands such as "take the mug from the table to the nightstand on the left of the bed". How do we develop AI that can talk with people about the real world? There are several research problems to address: understanding what people are referring to in a 3D environment, how to interact with the world and what language people use to describe interactions, and parsing complex dialogue in the world involving multiple agents. My research addresses these problems and explores fundamental questions that underpin a human-level understanding of the connection between language and the visual world. Progress in this challenging problem area can have immense impact. By tackling this problem, we can enable robotic assistants that can communicate with us using natural language about the world we live in. We can also make 2D and 3D content creation easier through language-driven UIs. Another application is in educational settings where we can use the learned mapping between language and visual concepts to teach foreign language speakers in virtual environments about how these concepts are carved out in a new language. Imagine, for example, a virtual reality game where you can look through your living room and see all the objects described in Spanish. Beyond these concrete applications, progress towards jointly understanding language and the physical world will form a foundation for general artificial intelligence, and is one ingredient for helping to combine machine perception and symbolic reasoning, a long-standing grand challenge problem in AI. Progress in this area will reinforce the strength and global leadership of Canadian academic institutions on topics related to artificial intelligence and machine learning. This proposal aims to build a joint model of language and the world by making the following contributions: 1) Build an online interactive simulation platform in which multiplayer dialogue can happen alongside manipulation of a 3D environment. 2) Investigate representations that jointly model the world and language. 3) Leverage these representations to design algorithms that can link language to the physical world.