CAREER: Theoretical foundations for deep learning and large-scale AI models

生成式人工智能模型在各个领域都表现出了非凡的能力，产生了变革性的社会影响。然而，由于理论基础有限，特别是在敏感应用方面，它们的强大功能带来了巨大的挑战和风险。该项目的主要目标是为生成式AI模型（包括语言模型和扩散模型）建立理论基础。该项目将研究神经网络的能力和局限性，如变压器和ResNets在这些模型中，并开发技术来解释这些黑盒系统中隐式实现的算法。理论研究将利用各种学科，包括变分推理，抽样方法，高维统计，计算复杂性理论和强化学习理论。研究结果将提供有价值的理论见解，并促进安全利用流行的基础模型，如ChatGPT和DALLE。本项目将建立一个理论基础，阐明语言模型和扩散模型的能力和局限性。该项目将研究三种关键的学习模式：情境学习，生成建模和决策。对于上下文学习，该项目将分析transformer可以隐式实现哪些算法，开发解释transformer中实现的算法的技术，并在元训练期间提供优化和泛化的保证。这个项目将推导出神经网络的条件，以表示基于扩散的生成建模的高维分数函数。对于决策，该项目将揭示如何对神经网络进行元训练以近似强盗和强化学习算法，并研究将神经网络用作决策代理的方法。这些成果将指导跨学科AI模型的原则性设计和负责任的部署。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Generative AI models have shown remarkable capabilities across various domains, making a transformative societal impact. However, their powerful capabilities present substantial challenges and risks due to limited theoretical foundations, especially regarding sensitive applications. The primary objective of this project is to establish a theoretical foundation for generative AI models including language models and diffusion models. The project will examine the capabilities and limitations of neural networks such as transformers and ResNets within these models, and develop techniques to interpret the algorithms implicitly implemented in these black-box systems. The theoretical investigation will leverage a diverse range of subjects including variational inference, sampling methods, high-dimensional statistics, computational complexity theory, and reinforcement learning theory. The results will provide valuable theoretical insights and promote the safe utilization of prevailing foundation models such as ChatGPT and DALLE. This project will establish a theoretical foundation to elucidate the capabilities and limitations of language models and diffusion models. The project will investigate three key learning modalities: in-context learning, generative modeling, and decision making. For in-context learning, this project will analyze which algorithms transformers can implicitly implement, develop techniques to interpret the algorithms implemented in transformers, and provide guarantees on optimization and generalization during meta-training. This project will derive conditions for neural networks to represent high-dimensional score functions for diffusion-based generative modeling. For decision-making, the project will reveal how neural networks can be meta-trained to approximate bandit and reinforcement learning algorithms and investigate approaches to employing neural networks as decision-making agents. The outcomes will guide principled design and responsible deployment of AI models across disciplines. The activities include graduate student training and new course developments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.