CAREER: Reinforcement Learning for Recursive Markov Decision Processes and Beyond

职业：递归马尔可夫决策过程及其他的强化学习

• 批准号: 2146563
• 负责人: Ashutosh Trivedi
• 金额: $ 59.66万
• 依托单位: University of Colorado at Boulder
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-05-01 至 2027-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2146563&HistoricalAwards=false
• 关键词: CAREER; Reinforcement; Learning; Recursive; Markov

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Reinforcement Learning (RL) is a sampling-based approach to optimization of Markov decision processes (MDPs), where agents rely on rewards to discover optimal solutions. When combined with powerful approximation schemes (e.g., deep neural networks), RL has been effective in highly complex tasks traditionally considered beyond reach of Artificial Intelligence. However, its sensitivity to the approximation parameters makes RL difficult to use (significant Machine Learning expertise is demanded of the programmer) and difficult to trust (manual approximations can invalidate guarantees). The vision of this project is to democratize RL by developing principled methodologies and powerful tools to improve the usability and trustworthiness of RL-based programming at scale. These research objectives are complemented by efforts to integrate the foundations of RL-based computability in CS education and to explore the role of RL-based programming in CS education.Approximation in RL is needed because RL algorithms with guaranteed convergence work on finite MDPs, and yet scale poorly. Approximation affects usability and trustworthiness. This proposal identifies two goals addressing both concerns: 1) to discover convergent RL beyond finite MDPs and 2) to develop abstraction-based approaches for RL with rigorous optimization guarantees. The success of the proposed approaches will be evaluated by their ability to handle systems at scale. The algorithms and datasets will be disseminated as open-source software. The proposed research makes fundamental contributions to three disciplines: formal methods, machine learning, and control theory; at the same time, it takes fundamental, concrete steps towards broadening participation in computing by making RL-based programming easier and more inclusive.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该奖项全部或部分由2021年美国救援计划法案（公法117-2）资助。强化学习（RL）是一种基于采样的马尔可夫决策过程（MDP）优化方法，其中代理依赖于奖励来发现最佳解决方案。当与强大的近似方案（例如，深度神经网络），RL在传统上被认为超出人工智能范围的高度复杂任务中一直很有效。然而，它对近似参数的敏感性使得RL难以使用（程序员需要大量的机器学习专业知识）并且难以信任（手动近似可能使保证无效）。该项目的愿景是通过开发原则性的方法和强大的工具来提高基于RL的编程的可用性和可信度，从而使RL民主化。这些研究目标的补充，努力整合的基础RL为基础的可计算性在CS教育和探索的作用RL为基础的编程在CS education.Approximation在RL是必要的，因为RL算法保证收敛工作有限的MDPs，但规模差。近似性影响可用性和可信度。该提案确定了两个目标，解决这两个问题：1）发现收敛RL超越有限的MDPs和2）开发基于抽象的方法，RL具有严格的优化保证。所提出的方法的成功将根据其大规模处理系统的能力来评估。算法和数据集将作为开放源码软件传播。 该研究计划对形式化方法、机器学习和控制理论这三个学科做出了重要贡献，同时，通过使基于RL的编程变得更容易和更具包容性，为扩大计算参与度迈出了重要的具体步骤。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

The Octatope Abstract Domain for Verification of Neural Networks.

用于验证神经网络的八位位抽象域。

• 发表时间: 2023
• 期刊: Formal Methods. FM 2023.
• 作者: Bak, S.;Dohmen, T.;Subramani, K.;Trivedi, A.;Velasquez, A.;Wojciechowski, P.
• 通讯作者: Wojciechowski, P.

LTL-Based Non-Markovian Inverse Reinforcement Learning

• DOI: 10.5555/3545946.3599102
• 发表时间: 2021-10
• 作者: Mohammad Afzal;Sankalp Gambhir;Ashutosh Gupta;S. Krishna;Ashutosh Trivedi;Alvaro Velasquez

Mungojerrie: Linear-Time Objectives in Model-Free Reinforcement Learning.

Mungojerrie：无模型强化学习中的线性时间目标。

• 发表时间: 2023
• 期刊: Tools and Algorithms for the Construction and Analysis of Systems. TACAS 2023.
• 作者: Hahn, Ernst Moritz;Perez, Mateo;Schewe, Sven;Somenzi, Fabio;Trivedi, Ashutosh;Wojtczak, Dominik
• 通讯作者: Wojtczak, Dominik

Recursive Reinforcement Learning

递归强化学习

• 发表时间: 2022
• 期刊: Conference on Neural Information Processing Systems (NeurIPS 2022
• 作者: Hahn, Ernst Moritz;Perez, Mateo;Schewe, Sven;Somenzi, Fabio;Trivedi, Ashutosh;Wojtczak, Dominik
• 通讯作者: Wojtczak, Dominik

Optimal Repair for Omega-Regular Properties

Omega-Regular 特性的最佳修复

• 发表时间: 2022
• 期刊: Automated Technology for Verification and Analysis. ATVA 2022.
• 作者: Dave, V.;Krishna, S.;Murali, V.;Trivedi, A.
• 通讯作者: Trivedi, A.