Assessing Discourse Relations in Language Generation from GPT-2
Assessing Discourse Relations in Language Generation from GPT-2
复制标题
DOI：
10.18653/v1/2020.inlg-1.8
复制
发表时间：
2020-04
期刊：
影响因子：
--
通讯作者：
Wei-Jen Ko;Junyi Jessy Li
中科院分区：
文献类型：
--
作者：
Wei-Jen Ko;Junyi Jessy Li
NLP的最新进展归功于大规模预训练语言模型的出现。特别是GPT-2，由于其从左到右的语言建模目标，它适合于生成任务，但其生成的文本的语言质量在很大程度上仍未被探索。我们的工作在理解GPT-2的输出语篇连贯性方面迈出了一步。我们对GPT-2在有机生成和微调情景下输出的显性话语关系的有效性进行了全面的研究。结果表明，GPT-2并不总是生成包含有效的话语关系的文本，然而，它的文本是更符合人类的期望在微调的情况下。我们提出了一个解耦的策略，以减轻这些问题，并强调显式建模话语信息的重要性。
Recent advances in NLP have been attributed to the emergence of large-scale pre-trained language models. GPT-2, in particular, is suited for generation tasks given its left-to-right language modeling objective, yet the linguistic quality of its generated text has largely remain unexplored. Our work takes a step in understanding GPT-2’s outputs in terms of discourse coherence. We perform a comprehensive study on the validity of explicit discourse relations in GPT-2’s outputs under both organic generation and fine-tuned scenarios. Results show GPT-2 does not always generate text containing valid discourse relations; nevertheless, its text is more aligned with human expectation in the fine-tuned scenario. We propose a decoupled strategy to mitigate these problems and highlight the importance of explicitly modeling discourse information.