Revisiting Simple Neural Probabilistic Language Models-猫眼课题宝

DOI：

10.18653/v1/2021.naacl-main.407

发表时间：

2021-04

期刊：

ArXiv

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Simeng Sun;Mohit Iyyer

语言建模的最新进展不仅是由神经架构的进步推动的，而且还通过硬件和优化的改进来推动。在本文中，我们重新审视 Bengio 等人的神经概率语言模型（NPLM）。 (2003)，它简单地连接固定窗口内的单词嵌入，并将结果通过前馈网络传递以预测下一个单词。当扩展到现代硬件时，该模型（尽管有许多限制）在字级语言模型基准测试中的表现比预期要好得多。我们的分析表明，NPLM 比具有短输入上下文的基线 Transformer 实现了更低的复杂性，但难以处理长期依赖性。受此结果的启发，我们修改了 Transformer，用 NPLM 的局部串联层替换其第一个自注意力层，这导致三个单词级语言建模数据集的困惑度虽小但一致降低。

Recent progress in language modeling has been driven not only by advances in neural architectures, but also through hardware and optimization improvements. In this paper, we revisit the neural probabilistic language model (NPLM) of Bengio et al. (2003), which simply concatenates word embeddings within a fixed window and passes the result through a feed-forward network to predict the next word. When scaled up to modern hardware, this model (despite its many limitations) performs much better than expected on word-level language model benchmarks. Our analysis reveals that the NPLM achieves lower perplexity than a baseline Transformer with short input contexts but struggles to handle long-term dependencies. Inspired by this result, we modify the Transformer by replacing its first self-attention layer with the NPLM’s local concatenation layer, which results in small but consistent perplexity decreases across three word-level language modeling datasets.

Revisiting Simple Neural Probabilistic Language Models

文献摘要

相似文献