CAREER: Theory and Algorithms for Learning with Frozen Pretrained Models
职业:使用冻结的预训练模型进行学习的理论和算法
基本信息
- 批准号:2339978
- 负责人:
- 金额:$ 58.4万
- 依托单位:
- 依托单位国家:美国
- 项目类别:Continuing Grant
- 财政年份:2024
- 资助国家:美国
- 起止时间:2024-07-01 至 2029-06-30
- 项目状态:未结题
- 来源:
- 关键词:
项目摘要
Adapting pretrained models to new tasks and new data is crucial for the democratization of modern machine learning. Application of large pretrained models to customized tasks or new data is difficult since full retraining of the model is not possible due to the huge size and inaccesibility of the original training data. The inaccessibility and large size of the training data also renders traditional domain adaptation techniques impractical. To overcome this challenge there has been shift towards modular adaptation methods that enable the fine-tuning of "frozen" pretrained models with minimal or no modifications to their internal parameters. Despite the success of these methods in some practical applications, there is a gap in theoretical understanding of the factors affecting the effectiveness of finetuning of pretrained models. This project develops a systematic framework that will lay theoretical foundations and lead to rigorous fine-tuning principles. The project has the potential to impact a wide range of scientific fields and industries that rely on adaptation of pretrained machine learning models. The project's broader impact includes the establishment of educational initiatives for advancing STEM, contributing to the development of the future workforce in modern machine learning.The project's goal is to establish a unified theory and devise new algorithms with provable guarantees for the emerging paradigm of learning with frozen pretrained models. A mathematical framework will be developed to facilitate the theoretical analysis of the expressive power of frozen pretrained models under various adaptation and fine-tuning methods. The project will investigate three adaptation strategies: (1) parameter-efficient fine-tuning, which updates a minimal portion of the pretrained model's parameters while keeping the rest unchanged; (2) input/output processing, which involves modifying the input entering or the output produced by the pretrained models; and (3) model composition, which constructs a system of multiple pretrained models to address more complex tasks. In addition to analyzing these methods, the project aims to develop novel adaptation algorithms with provable performance guarantees.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
使预训练模型适应新任务和新数据对于现代机器学习的民主化至关重要。 将大型预训练模型应用于定制任务或新数据是困难的,因为由于原始训练数据的巨大规模和不可访问性,完全重新训练模型是不可能的。 训练数据的不可访问性和大尺寸也使得传统的领域自适应技术不切实际。为了克服这一挑战,人们转向了模块化自适应方法,这种方法可以对“冻结”的预训练模型进行微调,而对其内部参数进行最小或不进行修改。尽管这些方法在一些实际应用中取得了成功,但对影响预训练模型微调有效性的因素的理论理解存在差距。该项目开发了一个系统的框架,将奠定理论基础,并导致严格的微调原则。该项目有可能影响广泛的科学领域和行业,这些领域和行业依赖于预先训练的机器学习模型的适应。该项目的更广泛影响包括建立推进STEM的教育计划,为现代机器学习的未来劳动力的发展做出贡献。该项目的目标是建立一个统一的理论,并设计新的算法,为使用冻结预训练模型的新兴学习范式提供可证明的保证。将开发一个数学框架,以便于在各种适应和微调方法下对冻结预训练模型的表达能力进行理论分析。该项目将研究三种适应策略:(1)参数有效的微调,更新预训练模型参数的最小部分,同时保持其余部分不变;(2)输入/输出处理,包括修改输入或预训练模型产生的输出;(3)模型组合,构建多个预训练模型的系统,以解决更复杂的任务。除了分析这些方法外,该项目还旨在开发具有可证明的性能保证的新型自适应算法。该奖项反映了NSF的法定使命,并且通过使用基金会的知识价值和更广泛的影响审查标准进行评估,被认为值得支持。
项目成果
期刊论文数量(0)
专著数量(0)
科研奖励数量(0)
会议论文数量(0)
专利数量(0)
数据更新时间:{{ journalArticles.updateTime }}
{{
item.title }}
{{ item.translation_title }}
- DOI:
{{ item.doi }} - 发表时间:
{{ item.publish_year }} - 期刊:
- 影响因子:{{ item.factor }}
- 作者:
{{ item.authors }} - 通讯作者:
{{ item.author }}
数据更新时间:{{ journalArticles.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ monograph.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ sciAawards.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ conferencePapers.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ patent.updateTime }}
KANGWOOK LEE其他文献
KANGWOOK LEE的其他文献
{{
item.title }}
{{ item.translation_title }}
- DOI:
{{ item.doi }} - 发表时间:
{{ item.publish_year }} - 期刊:
- 影响因子:{{ item.factor }}
- 作者:
{{ item.authors }} - 通讯作者:
{{ item.author }}
相似国自然基金
Research on Quantum Field Theory without a Lagrangian Description
- 批准号:24ZR1403900
- 批准年份:2024
- 资助金额:0.0 万元
- 项目类别:省市级项目
基于isomorph theory研究尘埃等离子体物理量的微观动力学机制
- 批准号:12247163
- 批准年份:2022
- 资助金额:18.00 万元
- 项目类别:专项项目
Toward a general theory of intermittent aeolian and fluvial nonsuspended sediment transport
- 批准号:
- 批准年份:2022
- 资助金额:55 万元
- 项目类别:
英文专著《FRACTIONAL INTEGRALS AND DERIVATIVES: Theory and Applications》的翻译
- 批准号:12126512
- 批准年份:2021
- 资助金额:12.0 万元
- 项目类别:数学天元基金项目
基于Restriction-Centered Theory的自然语言模糊语义理论研究及应用
- 批准号:61671064
- 批准年份:2016
- 资助金额:65.0 万元
- 项目类别:面上项目
相似海外基金
CAREER: Structured Minimax Optimization: Theory, Algorithms, and Applications in Robust Learning
职业:结构化极小极大优化:稳健学习中的理论、算法和应用
- 批准号:
2338846 - 财政年份:2024
- 资助金额:
$ 58.4万 - 项目类别:
Continuing Grant
CAREER: Continual Learning with Evolving Memory, Soft Supervision, and Cross-Domain Knowledge - Foundational Theory and Advanced Algorithms
职业:利用进化记忆、软监督和跨领域知识进行持续学习——基础理论和高级算法
- 批准号:
2338506 - 财政年份:2024
- 资助金额:
$ 58.4万 - 项目类别:
Continuing Grant
CAREER: Theory for Dynamic Graph Algorithms
职业:动态图算法理论
- 批准号:
2238138 - 财政年份:2023
- 资助金额:
$ 58.4万 - 项目类别:
Continuing Grant
CAREER: Parallel Algorithms: Theory for Practice
职业:并行算法:理论实践
- 批准号:
2238358 - 财政年份:2023
- 资助金额:
$ 58.4万 - 项目类别:
Continuing Grant
CAREER: Developing a unified theory of descriptive combinatorics and local algorithms
职业:发展描述性组合学和局部算法的统一理论
- 批准号:
2239187 - 财政年份:2023
- 资助金额:
$ 58.4万 - 项目类别:
Continuing Grant
CAREER: Learning Kernels in Operators from Data: Learning Theory, Scalable Algorithms and Applications
职业:从数据中学习算子的内核:学习理论、可扩展算法和应用
- 批准号:
2238486 - 财政年份:2023
- 资助金额:
$ 58.4万 - 项目类别:
Continuing Grant
CAREER: Toward Real-Time, Constraint-Aware Control of Complex Dynamical Systems: from Theory and Algorithms to Software Tools
职业:实现复杂动力系统的实时、约束感知控制:从理论和算法到软件工具
- 批准号:
2238424 - 财政年份:2023
- 资助金额:
$ 58.4万 - 项目类别:
Standard Grant
CAREER: Efficient Uncertainty Quantification in Turbulent Combustion Simulations: Theory, Algorithms, and Computations
职业:湍流燃烧模拟中的高效不确定性量化:理论、算法和计算
- 批准号:
2143625 - 财政年份:2022
- 资助金额:
$ 58.4万 - 项目类别:
Continuing Grant
CAREER: Nonsmooth Control Systems for Societal Networks with Data-Assisted Feedback Loops: Theory and Algorithms
职业:具有数据辅助反馈环的社会网络的非平滑控制系统:理论和算法
- 批准号:
2305756 - 财政年份:2022
- 资助金额:
$ 58.4万 - 项目类别:
Continuing Grant
CAREER: Nonsmooth Control Systems for Societal Networks with Data-Assisted Feedback Loops: Theory and Algorithms
职业:具有数据辅助反馈环的社会网络的非平滑控制系统:理论和算法
- 批准号:
2144076 - 财政年份:2022
- 资助金额:
$ 58.4万 - 项目类别:
Continuing Grant