RI: Learning Language Semantics from Perceptual Context

这个项目开发了使用机器学习的系统，通过训练只与感知上下文配对的句子来构建自然语言的语义分析器。PI之前的研究开发了通过对带有正式意义表示的注释的句子进行训练来获得语义解析器的系统；然而，建立这样的注释语料库的要求限制了结果系统的范围和准确性。这个项目将这些方法扩展到更像人类孩子一样学习语言，只使用暴露在语境中的话语。为了暂时绕过现有计算机视觉和机器人系统的限制，该项目主要在模拟环境中研究这一问题。它使用Robocup足球模拟器作为探索语言习得的一个领域。使用从物理模拟器状态抽象描述的现有方法来构建感知上下文的符号表示。当从感知上下文而不是直接监督学习时，系统必须解决指称不确定性，即一个句子可能涉及当前环境的许多不同方面。因此，本项目设计、实现和评估了能够从仅与模糊监督配对的句子中学习的算法。在Robocup环境和其他应用中的实验中对所开发的技术的有效性进行了评估。开发的技术最终可以移植到真正的机器人上，允许在机器人学中整合语言和感知。通过增加我们对如何通过在语境中使用语言而获得语言的理解，该项目还应该提供对人类语言学习的洞察。

This project develops systems that use machine learning to construct semantic analyzers for natural language by training on sentences paired only with their perceptual context. The PI's previous research developed systems that acquire semantic parsers by training on sentences annotated with formal meaning representations; however, the demands of building such annotated corpora limit the scope and accuracy of the resulting systems. This project extends these methods to learn language more like a human child, using only exposure to utterances in context. In order to temporarily circumvent the limitations of existing computer-vision and robotic systems, the project primarily studies the problem in simulated environments. It uses the Robocup soccer simulator as one domain in which to explore language acquisition. Existing methods for abstracting a description from the physical simulator state are used to construct a symbolic representation of the perceptual context. When learning from perceptual context instead of direct supervision, a system must address referential uncertainty, i.e. a sentence may refer to a multitude of different aspects of the current environment.Consequently, this project designs, implements, and evaluates algorithms that can learn from sentences paired only with ambiguous supervision. The effectiveness of the techniques developed are evaluated in experiments in the Robocup environment and other applications. The techniques developed can eventually be ported to real robots, allowing for an integration of language and perception in robotics. By increasing our understanding of how language can be acquired from its use in context, the project should also provide insight into human language learning.