CPS: Medium: Collaborative Research: Certifiable reinforcement learning for cyber-physical systems

我们建议推广和证明用于计算机物理系统(CP)控制的强化学习算法的性能。广义地说，强化学习应用于物理系统是指从数据中做出预测，以控制系统以达到性能标准的极值。该项目将特别侧重于开发适用于混合和多智能体控制系统的理论和算法，即具有连续和离散元件的系统以及具有多个决策智能体的系统，这些系统普遍存在于CPS中，跨越时空尺度和应用领域。强化学习算法在CPS控制中的应用还不够成熟，不能保证控制性能。鉴于这些局限性，本项目旨在为强化学习算法的验证奠定理论和计算基础，从而使其能够以较高的置信度在社会上部署。本项目将验证在具有非经典动态和非经典代价的系统中计算最优控制策略的强化学习算法。为了实现这一目标，我们将把最初为纯连续系统设计的收敛算法推广到状态经历离散和连续转变的混合控制系统。此外，我们的具体目标是确保这种方法适用于社会规模的CP，在这种情况下，多个代理，其中一些可能是人类，直接与CP交互。这些算法将在三个试验台上进行实验验证，这三个试验台代表了CPS中出现的一系列混合和多代理现象。第一个试验台将通过模拟来测试我们的算法在社会规模的交通流网络上的性能。第二个试验台将考虑空中和地面移动机器人的不同团队，与人类合作伙伴合作，在桥梁和隧道等基础设施的大规模复制品上执行建设、检查和维护任务。第三个试验台将研究个人与执行动态移动和操作行为的远程遥控机器人之间的闭环交互。该项目还将与技术政策专家共同组织一个跨学科研讨会，研讨会的结果将成为由PIS举办的跨学科多校园研究生级别研讨会的基础。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

We propose to generalize and certify the performance of reinforcement learning algorithms for control of cyber-physical systems (CPS). Broadly speaking, reinforcement learning applied to physical systems is concerned with making predictions from data to control the system to extremize a performance criterion. The project will particularly focus on developing theory and algorithms applicable to hybrid and multi-agent control systems, that is, systems with continuous and discrete elements and systems with multiple decision-making agents, which are ubiquitous in CPS across spatiotemporal scales and application domains. Reinforcement learning algorithms are not yet mature enough to guarantee performance when applied to control of CPS. In light of these limitations, this project aims to lay the theoretical and computational foundation to certify reinforcement learning algorithms so that they may be deployed in society with high confidence.This project will certify reinforcement learning algorithms that compute optimal control policies in systems with non-classical dynamics and non-classical costs. To achieve this goal, we will generalize convergent algorithms originally designed for purely continuous systems to apply in hybrid control systems whose states undergo a mixture of discrete and continuous transitions. Moreover, we specifically aim to ensure this approach is applicable to societal-scale CPS in which multiple agents, some of which may be humans, interact directly with the CPS. These algorithms will be experimentally validated on three testbeds that represent a range of hybrid and multi-agent phenomena that arise in CPS. The first testbed will test the performance of our algorithms on societal-scale traffic flow networks via simulation. The second testbed will consider heterogeneous teams of aerial and terrestrial mobile robots collaborating with human partners to perform construction, inspection, and maintenance tasks on scale facsimiles of infrastructure like bridges, and tunnels. The third testbed will study the closed-loop interaction between individual humans and remote, teleoperated robots that perform dynamic locomotion and manipulation behaviors. This project will also co-organize an interdisciplinary workshop with technology policy experts, the results of which will form the basis for an interdisciplinary multi-campus graduate-level seminar run by the PIs.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.