Collaborative Research: RI:Medium:MoDL:Mathematical and Conceptual Understanding of Large Language Models

合作研究：RI:Medium:MoDL:大型语言模型的数学和概念理解

• 批准号: 2211779
• 负责人: Sanjeev Arora
• 金额: $ 80万
• 依托单位: Princeton University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2025-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2211779&HistoricalAwards=false
• 关键词: Collaborative; Research; RI; Medium; MoDL

Large language models (LLMs) have achieved unprecedented success in natural language processing (NLP). Since language models are being seen as a cornerstone of artificial intelligence in the near future, there is a need to be able to understand them, and to convey that understanding to regulators as well as the general public. These models are based on deep neural networks that are trained from vast quantities of text and have been demonstrated to be highly useful in performing tasks such as question answering, text classification, machine translation and summarization. Despite the huge empirical success, there is little understanding about their inner workings. This project seeks to bridge the gap by developing conceptual and mathematical understanding about training and using LLMs. The project will advance such understanding. The project will also seek to develop and disseminate instructional materials and draw on ideas from the project to impact ongoing programs at their institution to help increase participation in computing by individuals from underrepresented groups. The project has three components. (1) We will first build simplified generative models that capture the intrinsic structures of text, and analyze language models that are trained on texts from such generative models. (2) We then analyze why the learned language models can encode useful information that helps a wide range of downstream tasks. (3) Finally, we analyze and design new adaptation methods for downstream tasks with quantitative sample and computational efficiency guarantees. Education and outreach plans are integrated into this project: the investigators will develop a new introductory course in machine learning and disseminate instructional materials, mentor graduate and undergraduate students from underrepresented groups (through Princeton Freshman Scholars Institute, Stanford Summer Teacher Research Program, REU’s) and organize research workshops to promote conversations between the theoretical machine learning and NLP community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

大型语言模型（LLM）在自然语言处理（NLP）中取得了前所未有的成功。由于语言模型在不久的将来被视为人工智能的基石，因此需要能够理解它们，并将这种理解传达给监管机构和公众。这些模型基于深度神经网络，这些网络是从大量文本中训练出来的，并且已经被证明在执行问答、文本分类、机器翻译和摘要等任务时非常有用。尽管在经验上取得了巨大的成功，但人们对它们的内部运作却知之甚少。该项目旨在通过发展有关培训和使用LLM的概念和数学理解来弥合差距。该项目将促进这种理解。该项目还将寻求开发和传播教学材料，并利用该项目的想法来影响其机构正在进行的项目，以帮助提高代表性不足群体的个人对计算的参与。该项目有三个组成部分。(1)我们将首先构建简化的生成模型，以捕获文本的内在结构，并分析在这些生成模型的文本上训练的语言模型。(2)然后，我们分析了为什么学习的语言模型可以编码有用的信息，帮助广泛的下游任务。(3)最后，我们分析和设计了新的适应方法的下游任务的定量样本和计算效率的保证。教育和外联计划被纳入该项目：研究人员将开发一门新的机器学习入门课程，分发教学材料，指导来自代表性不足群体的研究生和本科生（通过普林斯顿大学新生学者研究所、斯坦福大学暑期教师研究项目、REU's）并组织研究研讨会，以促进理论机器学习和NLP社区之间的对话。该奖项反映了NSF的法定基金会的使命是履行其使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评价，被认为值得支持。

项目成果

A Kernel-Based View of Language Model Fine-Tuning

• DOI: 10.48550/arxiv.2210.05643
• 发表时间: 2022-10
• 作者: Sadhika Malladi;Alexander Wettig;Dingli Yu;Danqi Chen;Sanjeev Arora

What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning

情境学习——学习什么——情境中：解开任务识别和任务学习

• DOI: 10.18653/v1/2023.findings-acl.527
• 发表时间: 2023
• 期刊: Association for Computational Linguistics
• 作者: Pan, Jane;Gao, Tianyu;Chen, Howard;Chen, Danqi
• 通讯作者: Chen, Danqi