CRII: CPS: Towards a Model-Based Reinforcement Learning Approach for Safe Operation of Distributed Energy Systems

随着可再生能源在电网上的日益普及和实时用电数据的可用性，电网正在成为一个大规模复杂的网络物理系统（CPS），以满足消费者每天的用电需求。强化学习（RL）算法为这些CPS系统提供了一种更高效、更有效、更经济地将分布式能源无缝集成到传统电网中的方法。它还提供了大量潜在的资本投资成本和劳动力节约，以及对服务中断的更大弹性。本研究项目开发了一个基于模型的在线强化学习框架，以解决几类问题。首先，将能量CPS的控制建模为有限水平RL问题。其次，这个项目不是关注渐近收敛，而是关注最优有限时间性能。第三，虽然简单的学习算法可能会将能源CPS驱动到不安全的操作区域，从而冒着不必要的后果，但该项目开发了安全的RL算法，可以优化性能并尊重安全约束。第四，该项目利用能源CPS的物理特性来避免维度问题，通常与RL问题相关。最后，该项目开发了使用“上下文强盗”方法的顺序算法，用于学习消费者特定参数和自适应调度，以考虑消费者的使用情况。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

With the increasing penetration of renewables on the electric grid and ready availability of real-time data about electricity usage, the electric power grid is becoming a large-scale complex Cyber-Physical System (CPS) to meet consumer demands for electricity through the day, every day. Reinforcement learning (RL) algorithms offer these CPS systems an approach to seamlessly integrating distributed energy sources into the legacy electric grid more efficiently, effectively and affordably. It also offers significant potential savings in capital investment cost and labor, and greater resiliency to disruptions in service. This research project develops a framework for model-based online reinforcement learning to address several classes of problems. First, it models control of energy CPS as finite horizon RL problems. Second, instead of focusing on asymptotic convergence, this project focuses on optimal finite time performance. Third, while a simplistic learning algorithm might drive an energy CPS to an unsafe region of operations, thereby risking unwanted consequences, this project develops safe RL algorithms that optimize performance and respect safety constraints. Fourth, this project exploits the physical properties of the energy CPS to avoid the dimensionality problems, often associated with RL problems. Lastly, the project develops sequential algorithms using a "contextual bandits" approach for learning consumer specific parameters and adaptively scheduling to account for consumer usage.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.