RI: Medium: Tree-Structured Self-Supervised Modeling for Natural Language

这个项目促进了理解人类语言的节能和语言驱动的计算方法的发展。机器翻译和问题回答等基于文本的任务的最新进展是通过训练针对数十亿个单词的大规模神经网络模型来推动的。尽管这种暴力手段无疑是成功的，但它也有许多不利之处。随着培训和使用这些模型的计算需求越来越大，它们的碳足迹一直在稳步增加，它们的可及性已仅限于少数资金雄厚的公司和机构。这些模型没有明确地考虑语言的层次性，这是语言学中一个研究得很好的现象，研究人员认为，这是导致它们总体效率低下的原因之一，也降低了它们对最终用户的可解释性。该项目开发的技术不仅旨在使语言的计算模型更容易获得和更有效，而且还旨在改善翻译和摘要等文本生成任务的最新水平。该项目将新开发的方法整合到学术环境中，为计算机科学以外的本科生以及代表不足的社区提供重要的拓展。为了开发这种新的方法，该项目引入了神经体系结构，从原始文本中归纳出句法和语义相关的树结构，同时学习强大的基于向量的表示法，从而改善下游任务。这些模型结合了来自自我监督学习的见解，允许强大的表征学习，而不需要昂贵的人工努力，以及树形结构偏差。根据三个主要目标对所产生的方法进行评估：(1)实现单个体系结构内整个语言层次(即词、短语、句子和语篇级别单元)的表征学习；(2)提高训练和推理的计算和能量效率；以及(3)改进包括文档级翻译和文本摘要在内的长格式文本生成任务。这项研究旨在推动对可持续和可扩展的语言表示学习的研究，因此其成果包括公开发布的预培训模型和开源代码。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project promotes the development of energy-efficient and linguistically-motivated computational methods for understanding human language. Recent advances in text-based tasks such as machine translation and question answering have been fueled by training huge-scale neural network models on billions of words. While this brute-force approach has no doubt been successful, it also has many downsides. As the computational requirements for training and using these models grow larger and larger, their carbon footprints have been steadily increasing, and their accessibility has become limited to those at a few well-funded companies and institutions. These models do not explicitly consider the hierarchical nature of language, a well-studied phenomenon in linguistics, which the investigators believe contributes to their overall inefficiency and also reduces their interpretability to end users. The technologies developed in this project aim not only to make computational models of language more accessible and efficient, but also to improve the state of the art in text generation tasks such as translation and summarization. The project integrates the newly-developed methods into academic settings to provide significant outreach to undergraduates outside of computer science as well as in underrepresented communities.To develop this new methodology, the project introduces neural architectures that induce syntactic and semantically-relevant tree structures from raw text while simultaneously learning powerful vector-based representations that improve downstream tasks. These models combine insights from self-supervised learning, which allows for powerful representation learning without expensive manual effort, with a tree-shaped structural bias. The resulting methods are evaluated with respect to three major goals: (1) enabling representation learning of the entire linguistic hierarchy (i.e., words, phrases, sentences, and discourse-level units) within a single architecture; (2) improving computational and energy efficiency of training and inference; and (3) improving long-form text generation tasks including document-level translation and text summarization. This research effort aims to spur research into sustainable and scalable language representation learning, and as such its outputs include publicly-released pretrained models and open-sourced code.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.