RI: Small: Perceptually Grounded Learning of Instructional Language

这个项目正在开发一种方法，使计算机能够自动学习理解和生成人类语言的指令。传统的自然语言学习方法要求语言学专家费力地给大量的句子加上语法和含义的详细信息。在这个项目中，教学语言最初是通过简单地观察人类跟随其他人给出的指令来学习的。一旦系统从观察中学习得相当好，它也会积极参与学习过程，自己遵循人类给出的指令，或者向人类发出指令并观察他们的行为。目前正在评估这种方法在虚拟环境中（例如，在虚拟环境中）解释和生成英语导航说明的能力。“沿着走廊走，经过椅子后向左拐。”)。一种新的机器学习方法从人类追随者执行的结果动作中推断出句子的可能形式意义，然后使用现有的语言学习方法获得语言解释器和生成器。该学习系统正在一系列虚拟环境中进行评估，测试其遵循人类提供的自然语言指令以实现规定目标的能力，以及生成人类可以成功遵循的自然语言指令以找到特定目的地的能力。为这个项目开发的方法将有助于游戏和教育模拟中的虚拟代理的发展，这些虚拟代理可以学习解释和生成英语指令，并最终帮助开发能够通过观察学习解释人类语言指令的机器人。

This project is developing methods that allow a computer to automatically learn to understand and generate instructions in human language. Traditional approaches to natural-language learning require linguistic experts to laboriously annotate large numbers of sentences with detailed information about their grammar and meaning. In this project, instructional language is initially learned by simply observing humans following instructions given by other humans. Once the system has learned reasonably well from observation, it also actively participates in the learning process by following human-given instructions itself, or giving its own instructions to humans and observing their behavior. The approach is being evaluated on its ability to interpret and generate English instructions for navigating in a virtual environment (e.g. "Go down the hall and turn left after you pass the chair."). A novel machine learning method infers a probable formal meaning for a sentence from the resulting actions performed by a human follower, and then existing language-learning methods are used to acquire a language interpreter and generator. The learned system is being evaluated in a range of virtual environments, testing its ability to follow human-provided natural language instructions to achieve prescribed goals, as well as to generate natural language instructions that humans can successfully follow to find specific destinations. The methods developed for this project will contribute to the development of virtual agents in games and educational simulations that learn to interpret and generate English instructions, and eventually aid the development of robots that can learn to interpret human language instruction from observation.