RI: Small: Linguistic Structure in Neural Sequence Models

RI：小：神经序列模型中的语言结构

• 批准号: 1718846
• 负责人: Jason Eisner
• 金额: $ 39.5万
• 依托单位: Johns Hopkins University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-08-01 至 2021-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1718846&HistoricalAwards=false
• 关键词: RI; Small; Linguistic; Structure; Neural

Over the past 25 years, the field of artificial intelligence has made great strides in the ability to automatically analyze and generate sequential data. Much of this progress has come by building probabilistic models. For example, mathematical descriptions of how words are typically used in context are based on a scientific understanding of the relationships among letters, sounds, words, and phrases, thanks to the field of linguistics. Probabilistic models based on this understanding have allowed us to develop computational, data-driven methods for reasoning about the likely structure and meaning of sentences. In the same way, probabilistic models of sequences of events have led to computational methods for predicting the unfolding of future events and reconstructing the ordering of past ones. This project starts with sophisticated probabilistic models of linguistic structure and event sequences, and aims to make them more powerful, by using "deep learning" (neural networks) to increase their sensitivity to contextual effects. Deep learning has already recently had a revolutionary impact on artificial intelligence. This research will focus on using deep learning to enhance probabilistic models in settings where the model must discover structure that is not provided in its training data, such as the compositional units of language or the causal relations among events.The planned model design will not focus on hand-engineered features, but rather on broad representational choices. The overall architectures are motivated by certain basic notions that linguists and other modelers have found indispensable in their analyses of empirical data as follows: (1) stick-breaking processes that respect duality of patterning, the linguistic notion that a word's internal form is not necessarily related to its external usage but is governed by separate rules or by chance; (2) finite-state transducers that can capture local editing that transforms an input sequence into an output sequence; (3) context-free grammars that can model hierarchical structure to help explain word sequences; and (4) temporal point processes that can capture process intensity, where different events are competing to occur next, and combinations of earlier events combine to elevate or suppress the rates of later events. The project will infuse these probabilistic techniques with recurrent neural networks, in particular, long short-term memory (LSTM) networks. In some cases, exact inference in the resulting models will not be tractable, necessitating the design of Monte Carlo or variational approximations.

在过去的25年里，人工智能领域在自动分析和生成顺序数据的能力方面取得了长足的进步。这一进步在很大程度上是通过建立概率模型取得的。例如，语言学领域对单词在上下文中的典型使用方式的数学描述是基于对字母、声音、单词和短语之间关系的科学理解。基于这一理解的概率模型使我们能够开发出计算的、数据驱动的方法，用于推理句子的可能结构和意义。同样，事件序列的概率模型导致了预测未来事件的展开和重建过去事件的顺序的计算方法。这个项目从复杂的语言结构和事件序列的概率模型开始，旨在通过使用“深度学习”(神经网络)来提高它们对上下文影响的敏感度，从而使它们变得更加强大。深度学习最近已经对人工智能产生了革命性的影响。这项研究的重点是在模型必须发现训练数据中没有提供的结构的情况下，使用深度学习来增强概率模型，例如语言的组成单位或事件之间的因果关系。计划中的模型设计不会关注手工设计的特征，而是关注广泛的代表性选择。整体架构是由语言学家和其他模型师发现在他们的经验数据分析中不可或缺的某些基本概念所驱动的：(1)尊重模式二元性的折断过程，即一个词的内部形式不一定与其外部用法相关，而是受单独的规则或偶然支配的语言概念；(2)可以捕获将输入序列转换为输出序列的局部编辑的有限状态转换器；(3)可以对层次结构进行建模以帮助解释单词序列的上下文无关语法；以及(4)可以捕捉过程强度的时间点过程，在该过程中，不同的事件竞相发生，并且早期事件的组合组合可以提高或抑制后期事件的发生率。该项目将把这些概率技术注入递归神经网络，特别是长期短期记忆(LSTM)网络。在某些情况下，所得到的模型中的精确推论将不容易处理，这就需要设计蒙特卡罗或变分近似。

项目成果

A Deep Generative Model of Vowel Formant Typology

元音共振峰类型学的深层生成模型

• DOI: 10.18653/v1/n18-1004
• 发表时间: 2018
• 期刊: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT
• 作者: Cotterell, Ryan;Eisner, Jason
• 通讯作者: Eisner, Jason

Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP

• 发表时间: 2021-12
• 期刊: ArXiv
• 作者: Sabrina J. Mielke;Zaid Alyafeai;Elizabeth Salesky;Colin Raffel;Manan Dey;Matthias Gallé;Arun Raja;Chenglei Si;Wilson Y. Lee;Benoît Sagot;Samson Tan

On the Diachronic Stability of Irregularity in Inflectional Morphology

论屈折形态中不规则性的历时稳定性

• 发表时间: 2018
• 期刊: Computing Research Repository (arXiv
• 作者: Ryan Cotterell;Christo Kirov;Mans Hulden;Jason Eisner
• 通讯作者: Jason Eisner

Unsupervised Disambiguation of Syncretism in Inflected Lexicons

• DOI: 10.18653/v1/n18-2087
• 发表时间: 2018-06
• 作者: Ryan Cotterell;Christo Kirov;Sabrina J. Mielke;Jason Eisner

Spelling-Aware Construction of Macaronic Texts for Teaching Foreign-Language Vocabulary

用于外语词汇教学的马卡罗语文本的拼写感知构建

• DOI: 10.18653/v1/d19-1679
• 发表时间: 2019
• 期刊: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing
• 作者: Renduchintala, Adithya;Koehn, Philipp;Eisner, Jason
• 通讯作者: Eisner, Jason