RI: Small: Linguistic Structure in Neural Sequence Models

在过去的25年里，人工智能领域在自动分析和生成序列数据的能力方面取得了巨大的进步。这些进步大多来自于建立概率模型。例如，由于语言学领域的发展，对单词在上下文中如何典型使用的数学描述是基于对字母、声音、单词和短语之间关系的科学理解。基于这种理解的概率模型使我们能够开发计算的、数据驱动的方法来推理句子的可能结构和含义。以同样的方式，事件序列的概率模型导致了预测未来事件展开和重建过去事件顺序的计算方法。这个项目从语言结构和事件序列的复杂概率模型开始，旨在通过使用“深度学习”（神经网络）来提高它们对上下文影响的敏感性，使它们更加强大。深度学习最近已经对人工智能产生了革命性的影响。本研究将侧重于使用深度学习来增强概率模型的设置，其中模型必须发现其训练数据中未提供的结构，例如语言的组成单元或事件之间的因果关系。计划中的模型设计将不会专注于手工设计的特征，而是广泛的代表性选择。总体架构是由某些基本概念驱动的，语言学家和其他建模者在对经验数据的分析中发现，这些基本概念是不可或缺的：(1)尊重模式二元性的断条过程，即一个词的内部形式不一定与其外部用法相关，而是由单独的规则或偶然支配的语言学概念；(2)能够捕获将输入序列转换为输出序列的本地编辑的有限状态换能器；(3)上下文无关语法，可以对层次结构进行建模，以帮助解释单词序列；(4)可以捕捉过程强度的时间点过程，其中不同的事件正在竞争下一个事件的发生，以及早期事件的组合结合起来提高或抑制后期事件的发生率。该项目将把这些概率技术与循环神经网络，特别是长短期记忆（LSTM）网络结合起来。在某些情况下，结果模型中的精确推断将无法处理，因此需要设计蒙特卡罗或变分近似。

Over the past 25 years, the field of artificial intelligence has made great strides in the ability to automatically analyze and generate sequential data. Much of this progress has come by building probabilistic models. For example, mathematical descriptions of how words are typically used in context are based on a scientific understanding of the relationships among letters, sounds, words, and phrases, thanks to the field of linguistics. Probabilistic models based on this understanding have allowed us to develop computational, data-driven methods for reasoning about the likely structure and meaning of sentences. In the same way, probabilistic models of sequences of events have led to computational methods for predicting the unfolding of future events and reconstructing the ordering of past ones. This project starts with sophisticated probabilistic models of linguistic structure and event sequences, and aims to make them more powerful, by using "deep learning" (neural networks) to increase their sensitivity to contextual effects. Deep learning has already recently had a revolutionary impact on artificial intelligence. This research will focus on using deep learning to enhance probabilistic models in settings where the model must discover structure that is not provided in its training data, such as the compositional units of language or the causal relations among events.The planned model design will not focus on hand-engineered features, but rather on broad representational choices. The overall architectures are motivated by certain basic notions that linguists and other modelers have found indispensable in their analyses of empirical data as follows: (1) stick-breaking processes that respect duality of patterning, the linguistic notion that a word's internal form is not necessarily related to its external usage but is governed by separate rules or by chance; (2) finite-state transducers that can capture local editing that transforms an input sequence into an output sequence; (3) context-free grammars that can model hierarchical structure to help explain word sequences; and (4) temporal point processes that can capture process intensity, where different events are competing to occur next, and combinations of earlier events combine to elevate or suppress the rates of later events. The project will infuse these probabilistic techniques with recurrent neural networks, in particular, long short-term memory (LSTM) networks. In some cases, exact inference in the resulting models will not be tractable, necessitating the design of Monte Carlo or variational approximations.