Adaptive dynamic coordination of damping controllers through deep reinforcement and transfer learning

通过深度强化和迁移学习实现阻尼控制器的自适应动态协调

• 批准号: 2033910
• 负责人: Hector Pulgar
• 金额: $ 21万
• 依托单位: University of Tennessee Knoxville
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-09-01 至 2023-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2033910&HistoricalAwards=false
• 关键词: Adaptive; dynamic; coordination; damping; controllers

In the last decades, global environmental pollution, concerns with fossil fuel reserves, and advances in technology have led to actions that are transforming the power grid. Over the years, several states have adopted renewable portfolio standards and goals to increase electricity production from renewable sources such as solar and wind power. This has resulted in new grid behaviors in response to disturbances in the system. This phenomenon has created concerns about an increased risk of sustained oscillations that can cause poor electric service quality and can even lead to blackouts. A search for new effective control systems has created a new breed of controllers. Specifically, the design of new controllers has been studied for wind turbines, energy storage systems, and other emergent components. However, with this massive presence of non-standard controllers and the existing controllers in conventional power plants, there is an urgent need for coordination that would enhance the combined effect of all the controllers to avoid conflicting interactions among them. Presently, there is no coordination of this magnitude in either actual systems or theoretical studies. The problem is challenging and it requires adaptability as the grid is permanently changing during its operation. This project will add intelligence to the grid, and through a real-time adaptable coordination, it will diminish possibilities of blackouts by mitigating the unwanted oscillations in the system. The proposed adaptive controller coordination has tremendous potential to enable the current and forthcoming power grid with superior dynamic performance and stability. This research will help increase awareness of the importance of the power grid for our nation including the benefits and challenges of renewable energy. Pre-college students and teachers will be exposed to engineering principles and practical applications through participation in outreach programs. This project will transform the conventional notion of controller coordination to make it suitable for real-time control as well as adaptive to disturbances and operating conditions. This project will seek controller coordinating signals that will minimize the system oscillation energy. The physical concept of oscillation energy not only allows avoiding the use of arbitrary objective functions, but also serves as a mechanism to weight the importance of the different oscillation modes without having to target in advance the most critical ones. As preliminary work, the PIs have derived an analytical procedure for on/off controller coordination using the time integral of the oscillation energy and its sensitivity with respect to the controller gains. As this procedure is based on the linearization of the state-space model, matrices, eigenvalues, and sensitivities must be either calculated promptly after a disturbance or collected previously off-line for the most representative operating points and disturbances. To overcome this drawback, and to make the coordination feasible and practical, a deep reinforcement learning (DRL) framework is proposed that would make the coordination not only adaptive but also more effective. In this regard, this DRL framework will allow exploring both discrete and continuous coordinating signals. While a discrete signal can be seen as a controller's on/off mechanism, a continuous signal can be understood as a quantity that can scale up/down the controller gains within a specified range. Furthermore, as the system can be subjected to extreme disturbances, this project proposes the use of transfer learning so the DRL training can be transferred even if there are topological changes or severe operational changes such as generator outages or load rejections.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在过去的几十年里，全球环境污染、对化石燃料储量的担忧以及技术的进步导致了电网转型的行动。多年来，一些州已经采用了可再生能源组合标准和目标，以增加太阳能和风能等可再生能源的发电量。这导致了新的网格行为，以响应系统中的干扰。这种现象引起了人们对持续振荡风险增加的担忧，持续振荡可能导致电力服务质量差，甚至可能导致停电。 对新的有效控制系统的研究已经创造了一种新的控制器。具体而言，新的控制器的设计已经研究了风力涡轮机，储能系统和其他紧急组件。然而，随着非标准控制器和传统发电厂中现有控制器的大量存在，迫切需要协调，以增强所有控制器的组合效果，以避免它们之间的相互冲突。目前，在实际系统或理论研究中还没有这种量级的协调。这个问题是具有挑战性的，它需要适应性，因为电网在运行过程中会不断变化。该项目将为电网增加智能，并通过实时适应性协调，通过减轻系统中不必要的振荡来减少停电的可能性。所提出的自适应控制器协调具有巨大的潜力，使当前和未来的电网具有上级动态性能和稳定性。这项研究将有助于提高人们对电网对我们国家的重要性的认识，包括可再生能源的好处和挑战。大学预科学生和教师将通过参与外展计划接触工程原理和实际应用。该项目将改变传统的控制器协调的概念，使其适用于实时控制以及自适应干扰和操作条件。该项目将寻求控制器协调信号，以最大限度地减少系统振荡能量。振荡能量的物理概念不仅允许避免使用任意的目标函数，而且还用作加权不同振荡模式的重要性的机制，而不必提前瞄准最关键的振荡模式。作为初步工作，PI已经推导出一个分析程序，用于开/关控制器协调使用的时间积分的振荡能量和它的灵敏度相对于控制器增益。由于该过程基于状态空间模型的线性化，因此必须在扰动后立即计算矩阵、特征值和灵敏度，或者预先离线收集最具代表性的操作点和扰动。为了克服这个缺点，并使协调可行和实用，提出了一个深度强化学习（DRL）框架，使协调不仅自适应，而且更有效。在这方面，该DRL框架将允许探索离散和连续的协调信号。虽然离散信号可以被视为控制器的开/关机制，但连续信号可以被理解为可以在指定范围内按比例增加/减少控制器增益的量。此外，由于系统可能会受到极端干扰，因此该项目建议使用迁移学习，即使发生拓扑变化或严重的操作变化（如发电机停机或负载拒绝），DRL培训也可以迁移。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Branching Dueling Q-Network-Based Online Scheduling of a Microgrid With Distributed Energy Storage Systems

• DOI: 10.1109/tsg.2021.3103405
• 发表时间: 2021-05
• 期刊: IEEE Transactions on Smart Grid
• 影响因子: 9.6
• 作者: H. Shuai;F. Li;Héctor Pulgar-Painemal;Yaosuo Xue

Multi-task deep reinforcement learning for intelligent multi-zone residential HVAC control

• DOI: 10.1016/j.epsr.2020.106959
• 发表时间: 2020-11
• 期刊: Electric Power Systems Research
• 影响因子: 3.9
• 作者: Yan Du;F. Li;J. Munk;Kuldeep R. Kurte;O. Kotevska;Kadir Amasyali;H. Zandi

Data-driven adaptive dynamic coordination of damping controllers

数据驱动的阻尼控制器自适应动态协调

• 发表时间: 2022
• 期刊: North American Power Symposium
• 作者: F. Zelaya-Arrazabal;J. Liu;J. Zhao;H. Pulgar-Painemal;H. Silva-Saravia
• 通讯作者: H. Silva-Saravia

Supplementary Primary Frequency Control Through Deep Reinforcement Learning Algorithms

通过深度强化学习算法补充主频控制

• DOI: 10.1109/naps58826.2023.10318681
• 发表时间: 2023
• 期刊: North American Power Symposium
• 作者: Zelaya-Arrazabal, Francisco;Thacker, Timothy;Pulgar-Painemal, Héctor;Guo, Zhenping
• 通讯作者: Guo, Zhenping

Post-storm repair crew dispatch for distribution grid restoration using stochastic Monte Carlo tree search and deep neural networks

• DOI: 10.1016/j.ijepes.2022.108477
• 发表时间: 2023-01
• 期刊: International Journal of Electrical Power & Energy Systems
• 作者: H. Shuai;Fangxing Li;Buxin She;Xiaofei Wang;Jin Zhao