A Cognitive Regularizer for Language Modeling
A Cognitive Regularizer for Language Modeling
复制标题
用于语言建模的认知正则化器
DOI：
--
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Ryan Cotterell
中科院分区：
文献类型：
--
作者：
Jason Wei;Clara Meister;Ryan Cotterell
均匀信息密度（UID）假说认为，表现最佳的说话者往往会在语言信号中均匀地分布信息，该假说作为对某些句法、形态和韵律选择的一种解释，在心理语言学中受到了关注。在这项工作中，我们探讨UID假说是否可作为统计语言建模的一种归纳偏差来实施。具体而言，我们用一个对UID进行编码的正则化项来扩充用于训练语言模型的标准最大似然估计（MLE）目标。在对涵盖五个语系的十种语言进行的实验中，我们发现使用UID正则化始终能提高语言模型的困惑度，在训练数据有限时效果更显著。此外，通过对生成序列的分析，我们发现经过UID正则化的语言模型具有其他理想的特性，例如，它们生成的文本在词汇上更加多样。我们的研究结果不仅表明UID是语言建模的一种合理的归纳偏差，而且还利用现代自然语言处理工具对UID假说提供了一种替代性验证。
The uniform information density (UID) hypothesis, which posits that speakers behaving optimally tend to distribute information uniformly across a linguistic signal, has gained traction in psycholinguistics as an explanation for certain syntactic, morphological, and prosodic choices. In this work, we explore whether the UID hypothesis can be operationalized as an inductive bias for statistical language modeling. Specifically, we augment the canonical MLE objective for training language models with a regularizer that encodes UID. In experiments on ten languages spanning five language families, we find that using UID regularization consistently improves perplexity in language models, having a larger effect when training data is limited. Moreover, via an analysis of generated sequences, we find that UID-regularized language models have other desirable properties, e.g., they generate text that is more lexically diverse. Our results not only suggest that UID is a reasonable inductive bias for language modeling, but also provide an alternative validation of the UID hypothesis using modern-day NLP tools.
什么样的语言难以进行语言建模？
DOI：
10.18653/v1/p19-1491
发表时间：
2019
期刊：
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL
影响因子：
--
作者：
Mielke, Sebastian J.;Cotterell, Ryan;Gorman, Kyle;Roark, Brian;Eisner, Jason
通讯作者：
Eisner, Jason