RI: Medium: Provable Reinforcement Learning with Function Approximation and Neural Networks

强化学习（RL）是一种通用的、灵活的序列决策框架。现代强化学习通常涉及具有大量状态的实际问题，其中必须部署函数逼近来将知识从访问状态推广到未访问状态。函数逼近，特别是以深度神经网络的形式，是强化学习最近在机器人、自动驾驶汽车、商业管理和生产系统等领域取得实际成功的核心。然而，现有的关于强化学习的理论认识大多局限于状态数较少的问题，没有使用函数逼近，强化学习的理论与实践之间仍然存在很大的差距。该项目旨在通过识别和解决RL中持续存在的基本挑战来弥合这一差距。为了实现这一目标，本项目将开发一套全面的函数逼近强化学习的基本理论和方法，特别强调其对现代深度强化学习的适用性。具体地说，这个项目将以两个平行的推力进行。第一篇论文研究了一般函数近似下的无模型RL。这个推力将确定函数类的一般结构，其中RL问题是可处理的，为这些一般函数类设计新的可证明有效的算法，并解决具有挑战性的问题，如模型错误规范。这篇文章将进一步将这些结果与深度学习的表示、优化和泛化方面的最新进展结合起来。第二个推动力涉及基于模型的强化学习，以整合领域知识。该推力将首先开发一种通用的基于模型的RL方法，使用价值目标系统识别的思想。该推力还将开发用于可处理计算的方法的随机逼近变体，以及用于在基于策略模型的强化学习之前分析非策略数据的深度模型约简或特征学习方法。该项目的重要成果将是新的通用和可靠的强化学习算法，保证在计算和统计效率的广泛应用中表现良好。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Reinforcement Learning (RL) is a generic and flexible framework for sequential decision-making problems. Modern RL commonly engages practical problems with an enormous number of states, where function approximation must be deployed to generalize knowledge from the visited states to the unvisited ones. Function approximation, particularly in the form of deep neural networks, lies at the heart of the recent practical successes of RL in domains such as robotics, autonomous vehicles, business management, and production systems. However, most existing theoretical understanding of RL has been restricted to the problems with a small number of states without using function approximation, and a significant gap remains between theory and practice of RL. This project seeks to bridge this gap by identifying and addressing the fundamental challenges that are persistent in RL with function approximation.To accomplish this goal, this project will develop a comprehensive set of fundamental theory and methodologies for RL with function approximation, with a special emphasis on its applicability to modern deep RL. Concretely, this project will proceed with two parallel thrusts. The first thrust investigates model-free RL with general function approximation. This thrust will identify the general structure of the function classes where RL problems are tractable, design new provably efficient algorithms for those general function classes, and address the challenging issues such as model misspecification. This thrust will further integrate these results with recent advances in representation, optimization and generalization of deep learning. The second thrust concerns model-based RL to incorporate domain knowledge. This thrust will first develop a general-purpose model-based RL method using the idea of value-targeted system identification. This thrust will also develop stochastic-approximation variants of the methods for tractable computation, and deep model reduction or feature learning methods for analyzing off-policy data prior to on-policy model-based RL. Important outcomes of this project will be new general and reliable RL algorithms that are guaranteed to perform well for a wide range of applications with both computational and statistical efficiency.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.