Adaptive dynamic coordination of damping controllers through deep reinforcement and transfer learning

在过去的几十年里，全球环境污染、对化石燃料储量的担忧以及技术的进步导致了正在改变电网的行动。多年来，几个州采用了可再生能源组合标准和目标，以增加太阳能和风能等可再生能源的发电量。这导致了新的网格行为，以响应系统中的干扰。这种现象引发了人们对持续振荡风险增加的担忧，持续振荡可能导致糟糕的电力服务质量，甚至可能导致停电。对新的有效控制系统的探索创造了一种新型的控制器。具体地说，已经研究了风力涡轮机、储能系统和其他紧急部件的新控制器的设计。然而，随着常规电厂中非标准控制器和现有控制器的大量存在，迫切需要协调以增强所有控制器的综合效果，以避免它们之间的相互冲突。目前，无论是在实际系统中还是在理论研究中，都没有这种规模的协调。这个问题是具有挑战性的，它需要适应性，因为电网在其运行过程中是永久性的变化。该项目将为电网增加智能，通过实时、自适应的协调，它将通过减轻系统中不必要的振荡来减少停电的可能性。所提出的自适应控制器协调具有巨大的潜力，使当前和即将到来的电网具有优越的动态性能和稳定性。这项研究将有助于提高人们对电网对我国重要性的认识，包括可再生能源的好处和挑战。大学预科学生和教师将通过参加外展计划接触到工程原理和实际应用。该项目将改变传统的控制器协调的概念，使其适用于实时控制以及对干扰和运行条件的适应。该项目将寻求控制器协调信号，使系统振荡能量最小化。振荡能量的物理概念不仅允许避免使用任意目标函数，而且还作为一种机制来权衡不同振荡模式的重要性，而不必预先针对最关键的模式。作为初步工作，PI利用振荡能量的时间积分及其对控制器增益的灵敏度推导出了开/关控制器协调的分析方法。由于这一过程是基于状态空间模型的线性化，因此必须在扰动发生后迅速计算矩阵、特征值和灵敏度，或者预先离线收集最具代表性的工作点和扰动的灵敏度。为了克服这一缺陷，为了使协调变得可行和实用，提出了一种深度强化学习(DRL)框架，使协调不仅具有自适应性，而且更加有效。在这方面，这个DRL框架将允许探索离散和连续的协调信号。虽然离散信号可以被视为控制器的开/关机制，但连续信号可以被理解为可以在指定范围内放大/缩小控制器增益的量。此外，由于系统可能受到极端干扰，该项目建议使用转移学习，以便即使发生拓扑变化或严重的操作变化，如发电机停机或负载拒绝，也可以转移DRL培训。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

In the last decades, global environmental pollution, concerns with fossil fuel reserves, and advances in technology have led to actions that are transforming the power grid. Over the years, several states have adopted renewable portfolio standards and goals to increase electricity production from renewable sources such as solar and wind power. This has resulted in new grid behaviors in response to disturbances in the system. This phenomenon has created concerns about an increased risk of sustained oscillations that can cause poor electric service quality and can even lead to blackouts. A search for new effective control systems has created a new breed of controllers. Specifically, the design of new controllers has been studied for wind turbines, energy storage systems, and other emergent components. However, with this massive presence of non-standard controllers and the existing controllers in conventional power plants, there is an urgent need for coordination that would enhance the combined effect of all the controllers to avoid conflicting interactions among them. Presently, there is no coordination of this magnitude in either actual systems or theoretical studies. The problem is challenging and it requires adaptability as the grid is permanently changing during its operation. This project will add intelligence to the grid, and through a real-time adaptable coordination, it will diminish possibilities of blackouts by mitigating the unwanted oscillations in the system. The proposed adaptive controller coordination has tremendous potential to enable the current and forthcoming power grid with superior dynamic performance and stability. This research will help increase awareness of the importance of the power grid for our nation including the benefits and challenges of renewable energy. Pre-college students and teachers will be exposed to engineering principles and practical applications through participation in outreach programs. This project will transform the conventional notion of controller coordination to make it suitable for real-time control as well as adaptive to disturbances and operating conditions. This project will seek controller coordinating signals that will minimize the system oscillation energy. The physical concept of oscillation energy not only allows avoiding the use of arbitrary objective functions, but also serves as a mechanism to weight the importance of the different oscillation modes without having to target in advance the most critical ones. As preliminary work, the PIs have derived an analytical procedure for on/off controller coordination using the time integral of the oscillation energy and its sensitivity with respect to the controller gains. As this procedure is based on the linearization of the state-space model, matrices, eigenvalues, and sensitivities must be either calculated promptly after a disturbance or collected previously off-line for the most representative operating points and disturbances. To overcome this drawback, and to make the coordination feasible and practical, a deep reinforcement learning (DRL) framework is proposed that would make the coordination not only adaptive but also more effective. In this regard, this DRL framework will allow exploring both discrete and continuous coordinating signals. While a discrete signal can be seen as a controller's on/off mechanism, a continuous signal can be understood as a quantity that can scale up/down the controller gains within a specified range. Furthermore, as the system can be subjected to extreme disturbances, this project proposes the use of transfer learning so the DRL training can be transferred even if there are topological changes or severe operational changes such as generator outages or load rejections.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.