CRII: CPS: Towards a Model-Based Reinforcement Learning Approach for Safe Operation of Distributed Energy Systems

随着可再生能源在电网中的渗透率越来越高，以及有关用电量的实时数据的可用性，电网正在成为一个大规模复杂的网络物理系统（CPS），以满足消费者每天对电力的需求。 强化学习（RL）算法为这些CPS系统提供了一种方法，可以更高效、有效和经济地将分布式能源无缝集成到传统电网中。 它还可以显著节省资本投资成本和劳动力，并提高服务中断的弹性。该研究项目开发了一个基于模型的在线强化学习框架，以解决几类问题。首先，它将能量CPS的控制建模为有限时域RL问题。其次，而不是专注于渐近收敛，这个项目的重点是最佳的有限时间性能。第三，虽然简单化的学习算法可能会将能量CPS驱动到不安全的操作区域，从而冒着不必要的后果，但该项目开发了安全的RL算法，可以优化性能并尊重安全约束。第四，该项目利用能量CPS的物理特性来避免通常与RL问题相关的维度问题。 最后，该项目开发了顺序算法，使用“上下文强盗”的方法来学习消费者特定的参数和自适应调度，以考虑消费者的使用情况。该奖项反映了NSF的法定使命，并被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

With the increasing penetration of renewables on the electric grid and ready availability of real-time data about electricity usage, the electric power grid is becoming a large-scale complex Cyber-Physical System (CPS) to meet consumer demands for electricity through the day, every day. Reinforcement learning (RL) algorithms offer these CPS systems an approach to seamlessly integrating distributed energy sources into the legacy electric grid more efficiently, effectively and affordably. It also offers significant potential savings in capital investment cost and labor, and greater resiliency to disruptions in service. This research project develops a framework for model-based online reinforcement learning to address several classes of problems. First, it models control of energy CPS as finite horizon RL problems. Second, instead of focusing on asymptotic convergence, this project focuses on optimal finite time performance. Third, while a simplistic learning algorithm might drive an energy CPS to an unsafe region of operations, thereby risking unwanted consequences, this project develops safe RL algorithms that optimize performance and respect safety constraints. Fourth, this project exploits the physical properties of the energy CPS to avoid the dimensionality problems, often associated with RL problems. Lastly, the project develops sequential algorithms using a "contextual bandits" approach for learning consumer specific parameters and adaptively scheduling to account for consumer usage.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.