Adaptive dynamic coordination of damping controllers through deep reinforcement and transfer learning

在过去的几十年里，全球环境污染、对化石燃料储量的担忧以及技术进步导致了改变电网的行动。多年来，一些州采用了可再生能源组合标准和目标，以增加太阳能和风能等可再生能源的发电量。这导致了新的网格行为，以响应系统中的干扰。这种现象引起了人们对持续振荡风险增加的担忧，这种振荡可能导致电力服务质量差，甚至可能导致停电。对新型有效控制系统的探索创造了一种新型的控制器。具体而言，研究了风力涡轮机、储能系统和其他紧急组件的新控制器设计。然而，在常规电厂中大量存在非标准控制器和现有控制器的情况下，迫切需要进行协调，以增强所有控制器的综合效果，避免它们之间的相互冲突。目前，无论是在实际系统还是理论研究中，都没有这种量级的协调。这个问题是具有挑战性的，它需要适应性，因为电网在运行过程中是不断变化的。该项目将为电网增加智能，并通过实时适应性协调，减少系统中不必要的振荡，从而减少停电的可能性。所提出的自适应控制器协调具有巨大的潜力，可以使当前和未来的电网具有优越的动态性能和稳定性。这项研究将有助于提高人们对电网对我们国家的重要性的认识，包括可再生能源的好处和挑战。大学预科学生和教师将通过参与外展项目接触到工程原理和实际应用。该项目将改变传统的控制器协调概念，使其适合于实时控制以及自适应干扰和操作条件。本课题将寻求使系统振荡能量最小化的控制器协调信号。振荡能量的物理概念不仅可以避免使用任意的目标函数，而且可以作为一种机制来权衡不同振荡模式的重要性，而不必事先针对最关键的模式。作为初步工作，pi已经推导出了一个分析程序，用于使用振荡能量的时间积分及其灵敏度相对于控制器增益的开关控制器协调。由于该过程基于状态空间模型的线性化，因此必须在扰动发生后立即计算矩阵、特征值和灵敏度，或者事先离线收集最具代表性的工作点和扰动。为了克服这一缺点，并使协调可行和实用，提出了一种深度强化学习（DRL）框架，使协调不仅自适应而且更有效。在这方面，该DRL框架将允许探索离散和连续协调信号。离散信号可以看作是控制器的开/关机制，而连续信号可以理解为可以在指定范围内放大/缩小控制器增益的量。此外，由于系统可能受到极端干扰，该项目提出使用迁移学习，因此即使存在拓扑变化或严重的操作变化（如发电机停机或负载拒绝），也可以转移DRL训练。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

In the last decades, global environmental pollution, concerns with fossil fuel reserves, and advances in technology have led to actions that are transforming the power grid. Over the years, several states have adopted renewable portfolio standards and goals to increase electricity production from renewable sources such as solar and wind power. This has resulted in new grid behaviors in response to disturbances in the system. This phenomenon has created concerns about an increased risk of sustained oscillations that can cause poor electric service quality and can even lead to blackouts. A search for new effective control systems has created a new breed of controllers. Specifically, the design of new controllers has been studied for wind turbines, energy storage systems, and other emergent components. However, with this massive presence of non-standard controllers and the existing controllers in conventional power plants, there is an urgent need for coordination that would enhance the combined effect of all the controllers to avoid conflicting interactions among them. Presently, there is no coordination of this magnitude in either actual systems or theoretical studies. The problem is challenging and it requires adaptability as the grid is permanently changing during its operation. This project will add intelligence to the grid, and through a real-time adaptable coordination, it will diminish possibilities of blackouts by mitigating the unwanted oscillations in the system. The proposed adaptive controller coordination has tremendous potential to enable the current and forthcoming power grid with superior dynamic performance and stability. This research will help increase awareness of the importance of the power grid for our nation including the benefits and challenges of renewable energy. Pre-college students and teachers will be exposed to engineering principles and practical applications through participation in outreach programs. This project will transform the conventional notion of controller coordination to make it suitable for real-time control as well as adaptive to disturbances and operating conditions. This project will seek controller coordinating signals that will minimize the system oscillation energy. The physical concept of oscillation energy not only allows avoiding the use of arbitrary objective functions, but also serves as a mechanism to weight the importance of the different oscillation modes without having to target in advance the most critical ones. As preliminary work, the PIs have derived an analytical procedure for on/off controller coordination using the time integral of the oscillation energy and its sensitivity with respect to the controller gains. As this procedure is based on the linearization of the state-space model, matrices, eigenvalues, and sensitivities must be either calculated promptly after a disturbance or collected previously off-line for the most representative operating points and disturbances. To overcome this drawback, and to make the coordination feasible and practical, a deep reinforcement learning (DRL) framework is proposed that would make the coordination not only adaptive but also more effective. In this regard, this DRL framework will allow exploring both discrete and continuous coordinating signals. While a discrete signal can be seen as a controller's on/off mechanism, a continuous signal can be understood as a quantity that can scale up/down the controller gains within a specified range. Furthermore, as the system can be subjected to extreme disturbances, this project proposes the use of transfer learning so the DRL training can be transferred even if there are topological changes or severe operational changes such as generator outages or load rejections.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.