Collaborative Research: CIF:Medium:Theoretical Foundations of Compositional Learning in Transformer Models

合作研究：CIF：Medium：Transformer 模型中组合学习的理论基础

• 批准号: 2403074
• 负责人: Dimitrios Papailiopoulos
• 金额: $ 50万
• 依托单位: University of Wisconsin-Madison
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-07-01 至 2028-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2403074&HistoricalAwards=false
• 关键词: Collaborative; Research; CIF; Medium; Theoretical

Large Language Models (LLMs) based on transformer architectures, such as GPT-4, Llama 2, and Claude 3, have demonstrated remarkable emergent capabilities in compositional reasoning, allowing them to tackle complex tasks by decomposing them into simpler intermediate steps. Examples to these tasks include text and code generation, basic arithmetic and problem solving, and answering complex questions. Despite these empirical advances, the underlying mechanics of these capabilities remain largely unexplored. This collaborative research project aims to investigate the theoretical foundations of compositional learning in transformer models, focusing on three key areas: model expressivity, statistical learning theory, and optimization, aiming to develop novel learning guarantees, algorithms, architectures, and design principles that significantly advance the development of more capable and interpretable Artificial Intelligence (AI) and LLM systems. The research findings will be incorporated into educational curricula, fostering a diverse community around transformers, compositional learning, and their applications. The project will also engage the broader public through workshops and outreach activities, promoting responsible AI practices and AI education for undergraduate and K-12 students.The first thrust will explore the expressive capacity of transformers augmented with loops, memory, and external tools, which are essential for compositional reasoning. The second thrust will examine the statistical properties of autoregressive training using compositional data to understand its limits, benefits, and ability to generalize to novel problem instances. This is expected to lead to new theories of compositional learning that will highlight the role of skill acquisition and composition. The third thrust will investigate the optimization principles of compositional learning with transformers. This research will shed light on the optimization landscape and identify techniques for more efficient training of transformers through compositional techniques.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

基于Transformer架构的大型语言模型（LLM），如GPT-4、Llama 2和Claude 3，在组合推理方面表现出了非凡的涌现能力，使它们能够通过将复杂任务分解为更简单的中间步骤来处理复杂任务。这些任务的示例包括文本和代码生成、基本算术和问题解决以及回答复杂问题。尽管取得了这些经验性进展，但这些能力的基本机制在很大程度上仍未得到探索。该合作研究项目旨在研究Transformer模型中组合学习的理论基础，重点关注三个关键领域：模型表达性，统计学习理论和优化，旨在开发新的学习保证，算法，架构和设计原则，显着推进更有能力和可解释的人工智能（AI）和LLM系统的开发。研究结果将被纳入教育课程，围绕变压器，组合学习及其应用培养一个多元化的社区。该项目还将通过研讨会和外展活动吸引更广泛的公众，促进负责任的人工智能实践和对本科生和K-12学生的人工智能教育。第一个推力将探索增强循环，记忆和外部工具的变形金刚的表达能力，这对组合推理至关重要。第二个重点将使用组合数据来检查自回归训练的统计特性，以了解其局限性，好处以及推广到新问题实例的能力。这有望导致新的组合学习理论，将突出技能的获得和组成的作用。第三个重点将研究使用transformer的组合学习的优化原则。这项研究将揭示优化景观，并通过组合技术确定更有效的变压器培训技术。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。