Collaborative Research: CIF: Medium: Statistical and Algorithmic Foundations of Efficient Reinforcement Learning
合作研究:CIF:媒介:高效强化学习的统计和算法基础
基本信息
- 批准号:2106778
- 负责人:
- 金额:$ 80万
- 依托单位:
- 依托单位国家:美国
- 项目类别:Continuing Grant
- 财政年份:2021
- 资助国家:美国
- 起止时间:2021-10-01 至 2025-09-30
- 项目状态:未结题
- 来源:
- 关键词:
项目摘要
As a data-driven paradigm for sequential decision making in unknown environments, Reinforcement Learning (RL) has received significant interest in recent years owing to its potential ability to solve difficult problems associated with future societal and scientific developments. However, the explosion of both model dimensionality and complexity in current and emerging applications exacerbates the challenge of achieving efficient RL in sample-starved situations, where data collection is expensive, time-consuming, or even high-stake, e.g., in clinical trials, online advertising, and autonomous systems. As a result, understanding and improving the sample and computational efficiencies of RL algorithms, sometimes under additional resource and system-level constraints, are rightly understood as critical to the successful deployment of RL in the future. In this project the PIs are involving students at all levels with diverse backgrounds in Electrical and Computer Engineering, and in Statistics, are developing education modules on RL to enrich the curriculum, and are co-organizing workshops and outreach activities to enable the broader dissemination of the project outcomes.Despite decades-long research efforts, the statistical and computational underpinnings of RL are still far from being well understood, especially when it comes to finite-sample and finite-time issues which are of crucial operational value. This research project is bridging the theory-practice gap of modern algorithmic approaches to RL. It is doing so by (i) characterizing fundamental limits for the sample and computations complexities in various RL settings, (ii) by developing performance guarantees and uncertainty quantification schemes, and (iii) by designing new computationally efficient algorithms that are provably near-optimal in terms of sample complexity in both single-agent and multi-agent settings. The expected outcomes will enable the trustworthy adoption of RL algorithms in sample-starved environments. The complementary expertise of the research team is being leveraged to enrich the statistical and algorithmic foundations of RL through model-, policy-, and value-based approaches. New efficient algorithms that rely on function approximation schemes are being developed in order to address the curse of dimensionality; the resulting techniques are intended to lead to non-asymptotic analysis tools that deal with the complicated statistical dependencies present in RL. This rich research agenda is expected to foster multidisciplinary efforts at the intersection of high-dimensional statistics, non-convex optimization, control theory, information theory, and machine learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
强化学习(RL)作为一种在未知环境中进行顺序决策的数据驱动范式,由于其解决与未来社会和科学发展相关的难题的潜在能力,近年来受到了极大的关注。然而,当前和新兴应用中模型维度和复杂性的爆炸加剧了在样本匮乏情况下实现高效RL的挑战,其中数据收集是昂贵的,耗时的,甚至是高风险的,例如,在临床试验、在线广告和自主系统中。因此,理解和提高RL算法的样本和计算效率,有时在额外的资源和系统级约束下,正确地理解为RL在未来成功部署的关键。在这个项目中,PI让不同背景的学生参与到各个级别的电气和计算机工程以及统计学中,正在开发RL的教育模块以丰富课程,并正在共同组织研讨会和外展活动,以便更广泛地传播项目成果。尽管数十年的研究努力,RL的统计和计算基础仍然远远没有得到很好的理解,特别是当涉及到有限样本和有限时间的问题,这是至关重要的业务价值。该研究项目正在弥合RL现代算法方法的理论与实践差距。它是通过(i)表征各种RL设置中的样本和计算复杂性的基本限制,(ii)通过开发性能保证和不确定性量化方案,以及(iii)通过设计新的计算效率高的算法,这些算法在单智能体和多智能体设置中的样本复杂性方面可证明接近最优。预期的结果将使RL算法在样本匮乏的环境中得到可靠的采用。研究团队的互补专业知识正在通过基于模型,策略和价值的方法来丰富RL的统计和算法基础。正在开发新的高效算法,依赖于函数近似方案,以解决维数灾难;由此产生的技术旨在导致非渐近分析工具,处理RL中存在的复杂的统计依赖关系。这个丰富的研究议程预计将促进在高维统计,非凸优化,控制理论,信息论和机器学习的交叉学科的努力。这个奖项反映了NSF的法定使命,并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。
项目成果
期刊论文数量(18)
专著数量(0)
科研奖励数量(0)
会议论文数量(0)
专利数量(0)
Tightening the Dependence on Horizon in the Sample Complexity of Q-Learning
- DOI:
- 发表时间:2021-02
- 期刊:
- 影响因子:0
- 作者:Gen Li;Changxiao Cai;Yuxin Chen;Yuantao Gu;Yuting Wei;Yuejie Chi
- 通讯作者:Gen Li;Changxiao Cai;Yuxin Chen;Yuantao Gu;Yuting Wei;Yuejie Chi
Fast Policy Extragradient Methods for Competitive Games with Entropy Regularization
- DOI:
- 发表时间:2021-05
- 期刊:
- 影响因子:0
- 作者:Shicong Cen;Yuting Wei;Yuejie Chi
- 通讯作者:Shicong Cen;Yuting Wei;Yuejie Chi
Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization
- DOI:10.1287/opre.2021.2151
- 发表时间:2020-07
- 期刊:
- 影响因子:0
- 作者:Shicong Cen;Chen Cheng;Yuxin Chen;Yuting Wei;Yuejie Chi
- 通讯作者:Shicong Cen;Chen Cheng;Yuxin Chen;Yuting Wei;Yuejie Chi
Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence
- DOI:10.1137/21m1456789
- 发表时间:2021-05
- 期刊:
- 影响因子:0
- 作者:Wenhao Zhan;Shicong Cen;Baihe Huang;Yuxin Chen;Jason D. Lee;Yuejie Chi
- 通讯作者:Wenhao Zhan;Shicong Cen;Baihe Huang;Yuxin Chen;Jason D. Lee;Yuejie Chi
Faster Last-iterate Convergence of Policy Optimization in Zero-Sum Markov Games
- DOI:10.48550/arxiv.2210.01050
- 发表时间:2022-10
- 期刊:
- 影响因子:0
- 作者:Shicong Cen;Yuejie Chi;S. Du;Lin Xiao
- 通讯作者:Shicong Cen;Yuejie Chi;S. Du;Lin Xiao
{{
item.title }}
{{ item.translation_title }}
- DOI:
{{ item.doi }} - 发表时间:
{{ item.publish_year }} - 期刊:
- 影响因子:{{ item.factor }}
- 作者:
{{ item.authors }} - 通讯作者:
{{ item.author }}
数据更新时间:{{ journalArticles.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ monograph.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ sciAawards.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ conferencePapers.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ patent.updateTime }}
Yuejie Chi其他文献
Settling the Sample Complexity of Model-Based Offline Reinforcement Learning
解决基于模型的离线强化学习的样本复杂度
- DOI:
10.48550/arxiv.2204.05275 - 发表时间:
2022 - 期刊:
- 影响因子:0
- 作者:
Gen Li;Laixi Shi;Yuxin Chen;Yuejie Chi;Yuting Wei - 通讯作者:
Yuting Wei
Memory-Limited stochastic approximation for poisson subspace tracking
泊松子空间跟踪的内存有限随机近似
- DOI:
- 发表时间:
2017 - 期刊:
- 影响因子:0
- 作者:
Liming Wang;Yuejie Chi - 通讯作者:
Yuejie Chi
Regularized blind detection for MIMO communications
MIMO 通信的正则盲检测
- DOI:
10.1109/isit.2010.5513407 - 发表时间:
2010 - 期刊:
- 影响因子:0
- 作者:
Yuejie Chi;Yiyue Wu;A. Calderbank - 通讯作者:
A. Calderbank
Principal subspace estimation for low-rank Toeplitz covariance matrices with binary sensing
具有二元感知的低秩 Toeplitz 协方差矩阵的主子空间估计
- DOI:
10.1109/acssc.2016.7869594 - 发表时间:
2016 - 期刊:
- 影响因子:0
- 作者:
H. Fu;Yuejie Chi - 通讯作者:
Yuejie Chi
Golay complementary waveforms for sparse delay-Doppler radar imaging
用于稀疏延迟多普勒雷达成像的 Golay 互补波形
- DOI:
- 发表时间:
2009 - 期刊:
- 影响因子:0
- 作者:
Yuejie Chi;Robert Calderbank;A. Pezeshki - 通讯作者:
A. Pezeshki
Yuejie Chi的其他文献
{{
item.title }}
{{ item.translation_title }}
- DOI:
{{ item.doi }} - 发表时间:
{{ item.publish_year }} - 期刊:
- 影响因子:{{ item.factor }}
- 作者:
{{ item.authors }} - 通讯作者:
{{ item.author }}
{{ truncateString('Yuejie Chi', 18)}}的其他基金
Federated Optimization over Bandwidth-Limited Heterogeneous Networks
带宽受限异构网络的联合优化
- 批准号:
2318441 - 财政年份:2023
- 资助金额:
$ 80万 - 项目类别:
Standard Grant
Collaborative Research: Towards a Theoretic Foundation for Optimal Deep Graph Learning
协作研究:为最优深度图学习奠定理论基础
- 批准号:
2134080 - 财政年份:2022
- 资助金额:
$ 80万 - 项目类别:
Continuing Grant
NSF Student Travel Grant for the Fifth Conference on Machine Learning and Systems (MLSys 2022)
第五届机器学习和系统会议 (MLSys 2022) 的 NSF 学生旅费补助金
- 批准号:
2219655 - 财政年份:2022
- 资助金额:
$ 80万 - 项目类别:
Standard Grant
Taming Nonlinear Inverse Problems: Theory and Algorithms
驯服非线性反问题:理论与算法
- 批准号:
2126634 - 财政年份:2021
- 资助金额:
$ 80万 - 项目类别:
Standard Grant
CIF: Small: Resource-Efficient Statistical Inference in Networked Environments
CIF:小型:网络环境中资源高效的统计推断
- 批准号:
2007911 - 财政年份:2020
- 资助金额:
$ 80万 - 项目类别:
Standard Grant
CIF: Medium: Collaborative Research: Theory of Optimization Geometry and Algorithms for Neural Networks
CIF:媒介:协作研究:神经网络优化几何理论和算法
- 批准号:
1901199 - 财政年份:2019
- 资助金额:
$ 80万 - 项目类别:
Standard Grant
EAGER-DynamicData: Subspace Learning From Binary Sensing
EAGER-DynamicData:从二进制感知中学习子空间
- 批准号:
1833553 - 财政年份:2018
- 资助金额:
$ 80万 - 项目类别:
Standard Grant
CIF: Small: Inverse Methods for Parametric Mixture Models
CIF:小:参数混合模型的逆方法
- 批准号:
1826519 - 财政年份:2018
- 资助金额:
$ 80万 - 项目类别:
Standard Grant
CIF: Medium: Collaborative Research: Nonconvex Optimization for High-Dimensional Signal Estimation: Theory and Fast Algorithms
CIF:中:协作研究:高维信号估计的非凸优化:理论和快速算法
- 批准号:
1806154 - 财政年份:2018
- 资助金额:
$ 80万 - 项目类别:
Continuing Grant
CAREER: Robust Methods for High-Dimensional Signal Processing under Geometric Constraints
职业:几何约束下高维信号处理的鲁棒方法
- 批准号:
1818571 - 财政年份:2018
- 资助金额:
$ 80万 - 项目类别:
Standard Grant
相似国自然基金
Research on Quantum Field Theory without a Lagrangian Description
- 批准号:24ZR1403900
- 批准年份:2024
- 资助金额:0.0 万元
- 项目类别:省市级项目
Cell Research
- 批准号:31224802
- 批准年份:2012
- 资助金额:24.0 万元
- 项目类别:专项基金项目
Cell Research
- 批准号:31024804
- 批准年份:2010
- 资助金额:24.0 万元
- 项目类别:专项基金项目
Cell Research (细胞研究)
- 批准号:30824808
- 批准年份:2008
- 资助金额:24.0 万元
- 项目类别:专项基金项目
Research on the Rapid Growth Mechanism of KDP Crystal
- 批准号:10774081
- 批准年份:2007
- 资助金额:45.0 万元
- 项目类别:面上项目
相似海外基金
Collaborative Research: CIF: Medium: Snapshot Computational Imaging with Metaoptics
合作研究:CIF:Medium:Metaoptics 快照计算成像
- 批准号:
2403122 - 财政年份:2024
- 资助金额:
$ 80万 - 项目类别:
Standard Grant
Collaborative Research: CIF-Medium: Privacy-preserving Machine Learning on Graphs
合作研究:CIF-Medium:图上的隐私保护机器学习
- 批准号:
2402815 - 财政年份:2024
- 资助金额:
$ 80万 - 项目类别:
Standard Grant
Collaborative Research: CIF: Small: Mathematical and Algorithmic Foundations of Multi-Task Learning
协作研究:CIF:小型:多任务学习的数学和算法基础
- 批准号:
2343599 - 财政年份:2024
- 资助金额:
$ 80万 - 项目类别:
Standard Grant
Collaborative Research: CIF: Small: Mathematical and Algorithmic Foundations of Multi-Task Learning
协作研究:CIF:小型:多任务学习的数学和算法基础
- 批准号:
2343600 - 财政年份:2024
- 资助金额:
$ 80万 - 项目类别:
Standard Grant
Collaborative Research: CIF-Medium: Privacy-preserving Machine Learning on Graphs
合作研究:CIF-Medium:图上的隐私保护机器学习
- 批准号:
2402817 - 财政年份:2024
- 资助金额:
$ 80万 - 项目类别:
Standard Grant
Collaborative Research: NSF-AoF: CIF: Small: AI-assisted Waveform and Beamforming Design for Integrated Sensing and Communication
合作研究:NSF-AoF:CIF:小型:用于集成传感和通信的人工智能辅助波形和波束成形设计
- 批准号:
2326622 - 财政年份:2024
- 资助金额:
$ 80万 - 项目类别:
Standard Grant
Collaborative Research: CIF-Medium: Privacy-preserving Machine Learning on Graphs
合作研究:CIF-Medium:图上的隐私保护机器学习
- 批准号:
2402816 - 财政年份:2024
- 资助金额:
$ 80万 - 项目类别:
Standard Grant
Collaborative Research: CIF: Medium: Snapshot Computational Imaging with Metaoptics
合作研究:CIF:Medium:Metaoptics 快照计算成像
- 批准号:
2403123 - 财政年份:2024
- 资助金额:
$ 80万 - 项目类别:
Standard Grant
Collaborative Research: NSF-AoF: CIF: Small: AI-assisted Waveform and Beamforming Design for Integrated Sensing and Communication
合作研究:NSF-AoF:CIF:小型:用于集成传感和通信的人工智能辅助波形和波束成形设计
- 批准号:
2326621 - 财政年份:2024
- 资助金额:
$ 80万 - 项目类别:
Standard Grant
Collaborative Research: CIF: Small: Versatile Data Synchronization: Novel Codes and Algorithms for Practical Applications
合作研究:CIF:小型:多功能数据同步:实际应用的新颖代码和算法
- 批准号:
2312872 - 财政年份:2023
- 资助金额:
$ 80万 - 项目类别:
Standard Grant