Microgrid energy scheduling under uncertain extreme weather: Adaptation from parallelized reinforcement learning agents
Microgrid energy scheduling under uncertain extreme weather: Adaptation from parallelized reinforcement learning agents
复制标题
DOI：
10.1016/j.ijepes.2023.109210
复制
发表时间：
2023
期刊：
影响因子：
--
通讯作者：
Avijit Das;Zhengbin Ni;Xiangnan Zhong
中科院分区：
文献类型：
--
作者：
Avijit Das;Zhengbin Ni;Xiangnan Zhong
微电网是整合可再生能源和提供无缝绿色电力以尽量减少碳足迹的有用解决方案。近年来，极端天气事件在世界范围内频繁发生，造成了重大的经济和社会损失。这些事件给微网能源调度问题带来了不确定性，增加了微网运行的挑战。传统的优化方法存在微网模型不确定和不可见事件的不准确性等问题。现有的基于强化学习（RL）的方法也受到有限的泛化和当需要随机公式来适应不确定性时不断增加的计算负担的阻碍。提出了一种基于概率事件的并行强化学习方法来处理微电网能量的不确定性。具体而言，采用多个局部学习代理以分布式方式与相关微电网环境交互，并将结果报告给全局代理，从而在极端事件期间在线优化微电网能源资源。对随机微电网能量优化问题进行了重新表述，使其包含所有可能的情形和概率。学习智能体的优势估计函数设计了反向扫描，将结果转移到价值函数更新过程中。两个仿真研究，随机优化和在线测试，进行了比较，与几种现有的RL方法。结果表明，PRL方法比基于经验重放的q学习方法和多智能体q学习方法的计算成本分别减少4倍和28倍，可实现高达20%的优化性能提升。
Microgrids are useful solutions for integrating renewable energy resources and providing seamless green electricity to minimize carbon footprint. In recent years, extreme weather events happened often worldwide and caused significant economic and societal losses. Such events bring uncertainties to the microgrid energy scheduling problems and increase the challenges of microgrid operation. Traditional optimization approaches suffer from the inaccuracy of the uncertain microgrid model and the unseen events. Existing reinforcement learning (RL) - based approaches are also hampered by the limited generalization and the increasing computational burden when stochastic formulations are required to accommodate the uncertainties. This paper proposes a new parallelized reinforcement learning (PRL) method based on the probabilistic events to handle the microgrid energy uncertainties. Specifically, several local learning agents are employed to interact with pertinent microgrid environments in a distributed manner and report outcomes to the global agent, which will optimize microgrid energy resources online during extreme events. The stochastic microgrid energy optimization problem is reformulated to include all possible scenarios with probabilities. The advantage estimate functions of learning agents are designed with a backward sweep to transfer the outcomes to the value function updating process. Two simulation studies, stochastic optimization and online testing, are performed to compare with several existing RL approaches. Results substantiate that the proposed PRL method can achieve up to 20% optimization performance improvement with 4 and 28 times less computation cost than Q-learning with experience replay and multi-agent Q-learning approaches, respectively.