RI: Small: Perceptually Grounded Learning of Instructional Language

该项目正在开发允许计算机自动学习理解和生成人类语言指令的方法。传统的自然语言学习方法需要语言专家费力地用语法和意义的详细信息来注释大量的句子。在这个项目中，教学语言最初是通过简单地观察人类遵循其他人给出的指令来学习的。 一旦系统从观察中学习得相当好，它也会通过遵循人类给出的指令，或向人类发出自己的指令并观察他们的行为，积极参与学习过程。目前正在评估这种方法解释和生成在虚拟环境中导航的英语指令的能力（例如，“沿着大厅走，经过椅子后左转。"). 一种新的机器学习方法从人类追随者执行的结果动作中推断句子的可能形式含义，然后使用现有的语言学习方法来获取语言解释器和生成器。 该学习系统正在一系列虚拟环境中进行评估，测试其遵循人类提供的自然语言指令以实现指定目标的能力，以及生成人类可以成功遵循以找到特定目的地的自然语言指令的能力。为该项目开发的方法将有助于在游戏和教育模拟中开发虚拟代理，学习解释和生成英语指令，并最终帮助开发能够从观察中学习解释人类语言指令的机器人。

This project is developing methods that allow a computer to automatically learn to understand and generate instructions in human language. Traditional approaches to natural-language learning require linguistic experts to laboriously annotate large numbers of sentences with detailed information about their grammar and meaning. In this project, instructional language is initially learned by simply observing humans following instructions given by other humans. Once the system has learned reasonably well from observation, it also actively participates in the learning process by following human-given instructions itself, or giving its own instructions to humans and observing their behavior. The approach is being evaluated on its ability to interpret and generate English instructions for navigating in a virtual environment (e.g. "Go down the hall and turn left after you pass the chair."). A novel machine learning method infers a probable formal meaning for a sentence from the resulting actions performed by a human follower, and then existing language-learning methods are used to acquire a language interpreter and generator. The learned system is being evaluated in a range of virtual environments, testing its ability to follow human-provided natural language instructions to achieve prescribed goals, as well as to generate natural language instructions that humans can successfully follow to find specific destinations. The methods developed for this project will contribute to the development of virtual agents in games and educational simulations that learn to interpret and generate English instructions, and eventually aid the development of robots that can learn to interpret human language instruction from observation.