Enhanced Power System Resiliency through Adaptive Automatic Remedial Action Selection using Multi-Agent Reinforcement Learning

电网一直面临着越来越多的运营挑战，包括自然灾害、极端天气条件以及为实现零碳足迹目标而进行的升级带来的日益复杂的挑战。这些因素向电力系统运营商提出了挑战，要求他们决定采取何种行动，以使大多数家庭保持电网供电，甚至避免大规模停电。即使控制中心的技术和实践随着时间的推移进行升级，电网故障和停电仍然会发生。当电网发生严重事故时，电力系统操作员必须遵循特定的程序来解决不利条件，以使系统恢复正常运行。考虑到上述挑战，有必要自动化选择最合适的补救行动的过程。在这种情况下，决策的速度对于避免大规模断电并将不利事件对电网设备的影响降至最低至关重要。在现代计算机科学方法的帮助下，响应时间的显著改善甚至完全自动化在实践中是可能的，例如机器学习算法，更具体地说，其学习方法的子组被称为强化学习。强化学习通过选择能带来最大收益的动作来获取知识，并已被证明可以解决复杂的问题，如训练机器人解决复杂的任务，甚至下复杂的游戏，如国际象棋和古老的围棋游戏。利用强化学习方法，本研究提出的解决方案允许我们通过选择减少潜在负面影响的动作，根据动作对消费者的影响来确定动作的优先顺序。如果关键设施断开的风险较低，则通过为潜在操作分配更高的优先级来实现这一点。为了充分发挥强化学习方法的优势，使用了多个“智能体”(决策者)。这些代理可以分布在不同的网格设施中，并执行本地控制操作。然而，为了确保电网的弹性，它们之间存在一定程度的中央协调。每个代理负责其自己的控制区域，在该区域中，它可以根据在代理的学习阶段以动作的形式提供的优先顺序指令来操作。通过这种方式，我们的目标是通过加入通过超高保真非线性模拟学习的关于电力系统的先验知识来增加代理的可信性。一旦代理人学到了足够的知识，就不再需要模拟，他们可以“在飞行中”做出决定。与一些电网控制中心使用的先前开发的优化方法相比，提出的解决方案的优势在于其超快的在线计算性能。因此，机器学习方法的灵活性使其能够对电力系统的安全进行详尽而快速的分析，同时考虑到对联网家庭的更广泛影响，这使得它们具有补充甚至取代现有解决方案的吸引力。该项目的工作不仅旨在推动上述方法的发展，而且还旨在建立概念验证工具，能够为电力系统运营商提供可操作的信息，以提高电力系统在考虑实际现实世界约束的情况下的弹性。这将通过与两家美国公用事业公司合作并使用他们的电网模型和测量来实现。如果成功，该项目的结果可能为电力系统运营的一种全新方法奠定基础。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Electrical power grids have been facing increased operational challenges, including those due to natural disasters, extreme weather conditions, and increasing complexity with the ongoing upgrades to achieve zero-carbon footprint goals. These factors challenge electrical power system operators in making decisions on which action to take in order to keep most of the households powered by the grid or even to avoid massive blackouts. Even though the control centers’ technology and their practices undergo upgrades over time, grid failures and blackouts still happen. When a severe event occurs on the grid, the power system operator has to follow specific procedures to resolve the unfavorable condition in order to bring the system back to normal operation. With the aforementioned challenges in mind, it becomes necessary to automate the process of choosing the most suitable remedial actions. In this case, the speed of decision-making is crucial to avoid massive disconnections and minimize the impact of unfavorable events on grid equipment. A significant improvement in response time and even full automation may become possible in practice with the help of modern computer science methods such as machine learning algorithms and, more specifically, its subgroup of learning methods that is known as reinforcement learning. Reinforcement learning acquires knowledge by choosing actions that provide the largest benefit and has been shown to solve complex problems, such as training robots to solve complicated tasks or even playing complex games such as chess and the ancient game of Go.Leveraging reinforcement learning methods, the proposed solution in this research allows us to prioritize an action according to its influence on consumers by choosing the action that reduces potential negative impacts. This is done by assigning higher priority to a potential action if the risk of disconnection of critical facilities is lower. To fully exploit the advantages of reinforcement learning methods, multiple “agents” (decision makers) are used. These agents can be distributed all across different grid facilities and perform local control actions. However, to assure grid resiliency, there is a degree of central coordination between them. Each agent is responsible for its own control area where it can operate according to the prioritized instructions that are provided in the form of actions during the agent’s learning phase. In such a way, we aim to increase the trustworthiness of the agents by incorporating prior knowledge about the power system that is learned via ultra-high-fidelity nonlinear simulations. Once the agents have learned enough, simulations are no longer needed and they can make their decisions "on the fly". The advantage of the proposed solution with respect to previously developed optimization methods in use in some grid control centers is their ultra-fast online computational performance. Thus, the flexibility of machine learning methods to perform exhaustive and fast analysis of a power system's security while considering a broader impact on the connected households makes them attractive to complement or even replace existing solutions. The work in this project aims not only to advance the development of the above mentioned methods but also to build proof-of-concept tools able to derive actionable information for power system operators to improve power system resiliency considering practical real-world constraints. This will be achieved by working together with two US utilities and using their grid models and measurements. If successful, the results of the project may lay the foundation of an entirely new approach for power system operation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.