Collaborative Research: RI:Medium:MoDL:Mathematical and Conceptual Understanding of Large Language Models

大语言模型(LLM)在自然语言处理(NLP)领域取得了前所未有的成功。由于语言模型在不久的将来被视为人工智能的基石，因此有必要能够理解它们，并将这种理解传达给监管机构和普通公众。这些模型基于从大量文本中训练的深度神经网络，已被证明在执行问题回答、文本分类、机器翻译和摘要等任务中非常有用。尽管取得了巨大的经验上的成功，但人们对它们的内部运作却知之甚少。该项目试图通过发展对培训和使用LLMS的概念和数学理解来弥合这一差距。该项目将促进这种理解。该项目还将寻求开发和传播教学材料，并借鉴该项目的想法，以影响其机构正在进行的计划，以帮助来自代表性不足群体的个人更多地参与计算。该项目有三个组成部分。(1)我们将首先构建捕捉文本内在结构的简化生成模型，并从这些生成模型中分析针对文本训练的语言模型。(2)然后分析为什么学习的语言模型能够编码有用的信息，从而帮助广泛的下游任务。(3)在保证定量采样和计算效率的前提下，分析和设计了新的下游任务自适应方法。教育和外展计划被整合到这个项目中：调查人员将开发一门新的机器学习入门课程，并传播教学材料，指导来自代表性不足群体的研究生和本科生(通过普林斯顿大学新生学者学院、斯坦福大学暑期教师研究计划，REU)，并组织研究研讨会，以促进理论机器学习和NLP社区之间的对话。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Large language models (LLMs) have achieved unprecedented success in natural language processing (NLP). Since language models are being seen as a cornerstone of artificial intelligence in the near future, there is a need to be able to understand them, and to convey that understanding to regulators as well as the general public. These models are based on deep neural networks that are trained from vast quantities of text and have been demonstrated to be highly useful in performing tasks such as question answering, text classification, machine translation and summarization. Despite the huge empirical success, there is little understanding about their inner workings. This project seeks to bridge the gap by developing conceptual and mathematical understanding about training and using LLMs. The project will advance such understanding. The project will also seek to develop and disseminate instructional materials and draw on ideas from the project to impact ongoing programs at their institution to help increase participation in computing by individuals from underrepresented groups. The project has three components. (1) We will first build simplified generative models that capture the intrinsic structures of text, and analyze language models that are trained on texts from such generative models. (2) We then analyze why the learned language models can encode useful information that helps a wide range of downstream tasks. (3) Finally, we analyze and design new adaptation methods for downstream tasks with quantitative sample and computational efficiency guarantees. Education and outreach plans are integrated into this project: the investigators will develop a new introductory course in machine learning and disseminate instructional materials, mentor graduate and undergraduate students from underrepresented groups (through Princeton Freshman Scholars Institute, Stanford Summer Teacher Research Program, REU’s) and organize research workshops to promote conversations between the theoretical machine learning and NLP community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.