Reconsidering the Past: Optimizing Hidden States in Language Models
Reconsidering the Past: Optimizing Hidden States in Language Models
复制标题
DOI：
10.18653/v1/2021.findings-emnlp.346
复制
发表时间：
2021-12
期刊：
影响因子：
--
通讯作者：
Davis Yoshida;Kevin Gimpel
中科院分区：
文献类型：
--
作者：
Davis Yoshida;Kevin Gimpel
提出了一种基于梯度的隐含状态优化(HSO)方法，用于提高变压器语言模型在推理时的性能。与动态评估(Krause等人，2018)类似，HSO计算语言模型分配给评估文本的对数概率的梯度，但使用它来更新缓存的隐藏状态，而不是模型参数。我们用预训练的Transformer-XL和GPT-2语言模型测试HSO，发现在困惑方面比WikiText103和PG-19数据集有改进，特别是在评估训练分布之外的模型时。我们还展示了在最近开发的基于提示的少发式评估设置中的收益，同样没有额外的参数或训练数据，从而展示了下游的适用性。
We present Hidden-State Optimization (HSO), a gradient-based method for improving the performance of transformer language models at inference time. Similar to dynamic evaluation (Krause et al., 2018), HSO computes the gradient of the log-probability the language model assigns to an evaluation text, but uses it to update the cached hidden states rather than the model parameters. We test HSO with pretrained Transformer-XL and GPT-2 language models, finding improvement on the WikiText103 and PG-19 datasets in terms of perplexity, especially when evaluating a model outside of its training distribution. We also demonstrate downstream applicability by showing gains in the recently developed prompt-based few-shot evaluation setting, again with no extra parameters or training data.