CAREER: Towards Real-world Reinforcement Learning

强化学习（RL）是数据驱动决策建模最重要的范式之一。近年来，强化学习取得了一些经验上的成功，比如在视频和棋盘游戏中，强化学习代理的表现超过了人类。然而，今天的许多经验RL算法通常需要许多训练示例来学习，并且可能产生不可靠的解决方案（例如，表现出灾难性失败的解决方案）。虽然在模拟器中训练强化学习代理时，这些问题通常不是问题，但在将强化学习部署到数据（包括人类反馈）昂贵且可靠性至关重要的现实问题时，它们会带来重大困难。这个项目的主要新颖之处在于开发新的强化学习算法，这些算法可以高效（从尽可能少的训练数据点）和可靠（避免高概率的灾难性故障）地学习。这种强化学习算法的发展可以将强化学习系统的应用从模拟扩展到现实世界的应用，在现实世界中，数据收集成本高昂，安全性至关重要。在自动驾驶方面，开发的技术可以使自动驾驶汽车安全地适应新的道路条件，减少错误。在生成式人工智能（AI）中，可以从丰富的人类反馈中学习的高效可靠的强化学习算法将使人类与人工智能更好地结合，使人工智能系统在人类的指导下可靠安全地改进。该项目的主要研究目标是通过在理论上和经验上推进强化学习技术来实现现实世界的强化学习。该项目的关键创新是通过利用特定的问题结构和丰富的人类反馈来开发安全高效的强化学习算法。该项目有三大重点。首先，该项目将建立规避风险的强化学习算法，这些算法被证明是正确的，并且可扩展到高维数据。其次，该项目将开发强化学习算法，可以利用常见的问题特定结构来提高样本效率。第三，该项目将为强化学习创建新的算法，除了标量奖励之外，还提供丰富的反馈（包括基于偏好的反馈和积极的演示）。除了关于算法进步的建议工作外，该项目将重点关注它们在现实世界问题中的部署，包括数据库查询优化和优化生成模型，如大型语言模型和扩散模型。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Reinforcement learning (RL) is one of the most important paradigms for modeling data-driven decision-making. Recent years have witnessed several empirical successes of RL, such as RL agents that outperform humans in video and board games. However, many empirical RL algorithms today often require many training examples to learn and can produce unreliable solutions (solutions that exhibit catastrophic failures, for example). While these issues are typically not problematic when training RL agents in simulators, they pose significant difficulties when deploying RL to real-world problems where data (including human feedback) is expensive, and reliability is essential. The main novelty of this project will be the development of new RL algorithms that can learn efficiently (from as few training data points as possible) and reliably (avoid catastrophic failures with high probability). The development of such RL algorithms can expand the applications of RL systems from simulation to real-world applications where data is expensive to collect and safety is critical. In autonomous driving, the developed technologies can make self-driving cars adapt to new road conditions safely by making fewer mistakes. In generative Artificial Intelligence (AI), efficient and reliable RL algorithms that can learn from rich human feedback will enable better human-AI alignment, making AI systems improve reliably and safely under human guidance.The main research goal of this project is to enable real-world RL by advancing RL techniques, theoretically and empirically. The critical innovation in the project is to develop safe and efficient RL algorithms by leveraging specific problem structures and rich human feedback. The project has three main thrusts. First, the project will establish risk-averse RL algorithms that are provably correct and scalable to high dimensional data. Second, the project will develop RL algorithms that can leverage common problem-specific structures for improved sample efficiency. Third, the project will create new algorithms for RL with rich feedback beyond scalar rewards (including preference-based feedback and positive demonstrations). In addition to the proposed work on algorithmic advancements, the project will focus on their deployment to real-world problems, including database query optimization and optimizing generative models such as Large Language Models and Diffusion Models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.