RI: Medium: Provable Reinforcement Learning with Function Approximation and Neural Networks

强化学习(RL)是一种通用的、灵活的序贯决策问题框架。现代RL通常涉及大量状态的实际问题，其中必须采用函数近似来将知识从已访问状态推广到未访问状态。函数逼近，特别是深度神经网络的形式，是RL最近在机器人、自动驾驶汽车、企业管理和生产系统等领域取得实际成功的核心。然而，现有的对相对论的理论理解大多局限于少数态的问题，没有采用函数逼近的方法，理论与实践之间存在着很大的差距。这个项目试图通过识别和解决函数逼近RL中持续存在的根本挑战来弥合这一差距。为了实现这一目标，该项目将发展一套全面的函数逼近RL的基本理论和方法，并特别强调其对现代深度RL的适用性。具体地说，这个项目将以两个平行的推力进行。第一个推力研究了一般函数逼近的无模型RL。这一推力将确定RL问题容易处理的函数类的一般结构，为这些一般函数类设计新的可证明有效的算法，并解决诸如模型错误指定等具有挑战性的问题。这一努力将进一步将这些结果与深度学习的表示、优化和推广方面的最新进展结合起来。第二个主旨涉及基于模型的RL，以整合领域知识。这将首先开发一个通用的基于模型的RL方法，使用以价值为目标的系统识别的思想。这一推力还将开发用于易处理计算的方法的随机近似变体，以及用于在基于策略模型的RL之前分析非策略数据的深度模型简化或特征学习方法。该项目的重要成果将是新的通用和可靠的RL算法，保证以计算和统计效率在广泛的应用程序中很好地执行。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Reinforcement Learning (RL) is a generic and flexible framework for sequential decision-making problems. Modern RL commonly engages practical problems with an enormous number of states, where function approximation must be deployed to generalize knowledge from the visited states to the unvisited ones. Function approximation, particularly in the form of deep neural networks, lies at the heart of the recent practical successes of RL in domains such as robotics, autonomous vehicles, business management, and production systems. However, most existing theoretical understanding of RL has been restricted to the problems with a small number of states without using function approximation, and a significant gap remains between theory and practice of RL. This project seeks to bridge this gap by identifying and addressing the fundamental challenges that are persistent in RL with function approximation.To accomplish this goal, this project will develop a comprehensive set of fundamental theory and methodologies for RL with function approximation, with a special emphasis on its applicability to modern deep RL. Concretely, this project will proceed with two parallel thrusts. The first thrust investigates model-free RL with general function approximation. This thrust will identify the general structure of the function classes where RL problems are tractable, design new provably efficient algorithms for those general function classes, and address the challenging issues such as model misspecification. This thrust will further integrate these results with recent advances in representation, optimization and generalization of deep learning. The second thrust concerns model-based RL to incorporate domain knowledge. This thrust will first develop a general-purpose model-based RL method using the idea of value-targeted system identification. This thrust will also develop stochastic-approximation variants of the methods for tractable computation, and deep model reduction or feature learning methods for analyzing off-policy data prior to on-policy model-based RL. Important outcomes of this project will be new general and reliable RL algorithms that are guaranteed to perform well for a wide range of applications with both computational and statistical efficiency.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.