Co-Optimizing Battery Storage for Energy Arbitrage and Frequency Regulation in Real-Time Markets Using Deep Reinforcement Learning
Co-Optimizing Battery Storage for Energy Arbitrage and Frequency Regulation in Real-Time Markets Using Deep Reinforcement Learning
复制标题
使用深度强化学习共同优化电池存储以实现实时市场中的能源套利和频率调节
作者：
Yushen Miao;Tianyi Chen;Shengrong Bu;Hao Liang;Zhu Han
电池储能系统（BESS）在消除与可再生能源发电相关的不确定性，保持电网稳定性和提高电网灵活性方面发挥着关键作用。在本文中，一个BESS是用来提供能源套利（EA）和频率调节（FR）服务的同时，以最大限度地提高其总收入的物理约束。EA和FR行动在不同的时间尺度上进行。多时间尺度问题被制定为两个嵌套马尔可夫决策过程（MDP）的子模型。该问题是一个复杂的决策问题，具有巨大的高维数据和不确定性（例如，电的价格）。因此，提出了一种新的协同优化方案来处理多时间尺度问题，并协调EA和FR服务。采用三重深度确定性策略梯度和探测噪声衰减（TDD-ND）方法来获得每个时间尺度上的最优策略。利用美国PJM监管市场的实时电价和监管信号数据进行了仿真。仿真结果表明，该方法的性能优于文献中研究的其他政策。
Battery energy storage systems (BESSs) play a critical role in eliminating uncertainties associated with renewable energy generation, to maintain stability and improve flexibility of power networks. In this paper, a BESS is used to provide energy arbitrage (EA) and frequency regulation (FR) services simultaneously to maximize its total revenue within the physical constraints. The EA and FR actions are taken at different timescales. The multitimescale problem is formulated as two nested Markov decision process (MDP) submodels. The problem is a complex decision-making problem with enormous high-dimensional data and uncertainty (e.g., the price of the electricity). Therefore, a novel co-optimization scheme is proposed to handle the multitimescale problem, and also coordinate EA and FR services. A triplet deep deterministic policy gradient with exploration noise decay (TDD–ND) approach is used to obtain the optimal policy at each timescale. Simulations are conducted with real-time electricity prices and regulation signals data from the American PJM regulation market. The simulation results show that the proposed approach performs better than other studied policies in literature.