Collaborative Research: CIF: Medium: Statistical and Algorithmic Foundations of Distributionally Robust Policy Learning

合作研究：CIF：媒介：分布式稳健政策学习的统计和算法基础

• 批准号: 2312205
• 负责人: Zhengyuan Zhou
• 金额: $ 40万
• 依托单位: New York University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2027-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2312205&HistoricalAwards=false
• 关键词: Collaborative; Research; CIF; Medium; Statistical

Efficient data-driven policy learning and deployment techniques are transforming many facets of our society as a result of their broad applicability in engineering, scientific and societal applications. Given the access to high-performance computing, the use of simulators and digital twins, for example, have emerged as practical alternatives to test and learn complex optimization policies. As a result, significant scholarly efforts have been devoted to this research area in the past decade. However, despite having made landmark progress, existing work in this area often makes a key (implicit) assumption; namely, that the environment in which the policy is trained will be the same as the environment in which the policy is deployed. Policies learned under this assumption can be fragile, as this assumption often does not hold in practical environments, either due to the simulator model specification or environment shifts. The goal of this project is to study statistical and algorithmic foundations for developing provably efficient robust policy learning in unknown environments, under a possibly misspecified generative model. The project studies comprehensive statistical and algorithmic foundations for distributionally robust policy learning in contextual bandits and reinforcement learning (RL) environments and develops statistically optimal and computationally efficient algorithms across a wide range of non-parametric distributional shifts. These provide a powerful framework for capturing model-agnostic environment changes, but at the same time, pose intellectual challenges as the unknown worst-case environment lies in an infinite-dimensional space. The presented program opens up several fundamental research directions that call for novel and principled developments. First, the project develops information-theoretic tools to understand the fundamental learning limits for distributionally robust policy learning and to characterize how the distributional uncertainty contributes to the difficulty of learning. Additionally, the project develops computationally efficient and statistically optimal estimation schemes for distributionally robust performance analysis of a given policy. Lastly, the project translates the efficiency gains in estimation due to learning a distributionally robust policy.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

由于其在工程，科学和社会应用中的广泛适用性，有效的数据驱动政策学习和部署技术正在改变我们社会的许多方面。考虑到获得高性能计算的访问，例如，模拟器和数字双胞胎的使用已成为测试和学习复杂优化策略的实用替代方法。结果，在过去十年中，已经致力于该研究领域的重大学术努力。但是，尽管取得了里程碑意义的进步，但在这一领域的现有工作通常是一个关键（隐性）假设。也就是说，对培训政策的环境将与部署政策的环境相同。在此假设下学习的策略可能是脆弱的，因为由于模拟器模型规范或环境变化，该假设通常不存在于实际环境中。该项目的目的是研究在可能不明的生成模型下，在未知环境中开发可证明有效的强大策略学习的统计和算法基础。 该项目研究在上下文匪徒和强化学习（RL）环境中为分布强大的策略学习（RL）环境中的综合统计和算法基础，并在广泛的非参数分布转移范围内开发了统计上最佳和计算有效的算法。这些为捕获模型不足的环境变化提供了一个强大的框架，但与此同时，构成智力挑战，因为未知最坏的环境在于无限维度的空间。提出的计划打开了几个基本的研究方向，这些方向需要新颖和有原则的发展。首先，该项目开发了信息理论工具，以了解分配强大的策略学习的基本学习限制，并表征分布不确定性如何有助于学习的困难。此外，该项目还开发了计算高效且统计上最佳的估计方案，以分配给定策略的稳健性能分析。最后，该项目将由于学习分配强大的政策而导致的估计效率提高。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子优点和更广泛的影响评估标准通过评估来获得支持的。