Collaborative Research: CIF: Medium: Learning to Control from Data: from Theory to Practice

数据驱动的决策在当今世界发挥着越来越重要的作用，例如从疫情应对到拼车优化。然而，从数据中学习最佳控制策略在离线和在线环境中都面临挑战：(A)(离线)不清楚如何最有效地利用先验收集的可用数据集，特别是当它不包括所有可能的感兴趣的情景时。(B)(在线)不清楚如何在可能代价高昂且不安全的情况下，通过与环境的最低限度互动来收集数据集。在应对这两个挑战的需要的推动下，本项目旨在提高强化学习(RL)在两种环境下的样本效率。此外，该项目计划纳入实践中所需的适应性和可信性。与这些研究项目相辅相成的活动包括培训未来的学术界、工业界和政府领导人，让他们掌握数据驱动的决策制定的基本技能。该项目的目标是开发新一代数据驱动的决策规则的理论和算法，以应对现代RL中的关键挑战。具体地说，研究议程的目的是(I)设计样本高效和计算高效的算法，用于在线和离线RL的函数逼近，以及(Ii)增强现有RL范例的适应性和可信度。为了实现第一个目标，提出在现有方法的基础上，结合神经网络忠实的不确定性量化方法，将对在线RL的乐观探索和对离线RL的悲观开发结合起来。为了实现第二个目标，建议在紧密样本复杂性特征的帮助下将模型选择纳入现有方法。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Data-driven decision-making is playing an increasingly critical role in today's world with examples ranging from epidemic response to ridesharing optimization. However, learning an optimal control policy from data faces challenges in both the offline and online settings: (a) (Offline) It is unclear how to most efficiently utilize the available dataset which was collected a priori, especially when it does not cover all possible scenarios of interest. (b) (Online) It is unclear how to collect a dataset through minimal interactions with the environment in situations where it may be costly and unsafe to do so. Driven by the need to address these two challenges, this project aims to improve the sample efficiency of reinforcement learning (RL) in both settings. In addition, the project plans to incorporate adaptivity and trustworthiness that are required in practice. Activities complementary to these research thrusts include the training of future leaders of academia, industry, and government by equipping them with fundamental skills in data-driven decision making.The goal of this project is to develop the theory and algorithms for a new generation of data-driven decision rules in order to address critical challenges in modern RL. Specifically, the research agenda aims (i) to design sample-efficient and computationally-efficient algorithms for online and offline RL with function approximation, and (ii) to enhance the adaptivity and trustworthiness of existing RL paradigms. To achieve the first goal, it is proposed to incorporate optimistic exploration for online RL and pessimistic exploitation for offline RL into existing approaches with the help of faithful uncertainty quantification for neural networks. To achieve the second goal, it is proposed to incorporate model selection into existing approaches with the help of tight sample complexity characterizations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.