CAREER: Building Next-Generation Language Models Based on Retrieval

大型语言模型(LMS)使自然语言处理领域发生了革命性的变化，在广泛的下游任务中实现了最先进的性能。尽管取得了成功，但这些学习管理系统是在海量文本数据上进行培训的，创建和运行的成本也很高。此外，它们本身就很难解释，难以根据不断变化的现实世界信息进行更新，并且可能会泄露用户的私人信息。这一提议寻求开发一种替代标准语言建模的范式：基于检索的语言模型，目的是减少培训和推理成本，同时提供更好的可解释性、适应性和私密性等好处。这项研究将通过开发本科和研究生自然语言处理课程、促进本科研究教育以及面向来自代表性不足社区的K-12学生和教师的新教学模块整合到教育中。本项目涉及包括培训、扩展、调整和使用基于检索的语言模型的全流程，并由四个部分组成，包括：(1)为基于检索的LMS建立一个通用的学习框架，并开发可扩展的算法以支持端到端的学习；(2)研究基于检索的LMS的缩放规律，并开发更好的数据量化以提高推理效率；(3)设计快速更新基于检索的LMS的方法，并使其适用于不可见和隐私敏感的领域；(4)设计有效的方法，在下游任务中使用基于检索的LMS。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Large language models (LMs) have revolutionized the field of natural language processing, achieving state-of-the-art performance in a wide range of downstream tasks. Despite the success, these LMs are trained on enormous amounts of text data and cost a massive amount to create and run. Additionally, they are inherently difficult to interpret, challenging to update with ever-changing real-world information, and may leak private user information. This proposal seeks to develop an alternative paradigm to standard language modeling: retrieval-based language models, with the aim of reducing training and inference costs while also providing benefits such as better interpretability, adaptability, and privacy. This research will be integrated into education through new teaching modules in developing undergraduate and graduate natural language processing courses, promoting education for undergraduate research, and outreach to K-12 students and teachers from underrepresented communities.This project addresses a full pipeline including training, scaling, adapting, and using retrieval-based language models and is organized into four components, including (1) building a general learning framework for retrieval-based LMs and developing scalable algorithms to support end-to-end learning; (2) investigating the scaling law of retrieval-based LMs and developing better data quantization to improve inference efficiency; (3) devising methods to quickly update retrieval-based LMs and adapt them to unseen and privacy-sensitive domains; (4) designing effective approaches to use retrieval-based LMs on downstream tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.