CPS: Small: Data-Driven Reinforcement Learning Control of Large CPS Networks using Multi-Stage Hierarchical Decompositions

目前，基于机器学习的大型复杂网络（如电力系统）实时控制在很大程度上受到了维数诅咒的瓶颈。即使是最简单的控制设计也需要复杂的数值来完成。当网络模型未知时，问题变得更加具有挑战性，因为需要适应额外的学习时间。这个项目将采取一个新的立场来解决这个问题，并开发一套分层或嵌套的基于机器学习的方案，利用网络动力学中各种形式的物理冗余来学习其行为的最重要特征，而不是浪费时间学习可能只会少量提高闭环性能的次要特征。这种选择性学习方法将使学习时间减少几个数量级，使实时控制更易于处理和实现。产品将包括适用于各种基于机器学习的控制的数值算法。在社会影响方面，该项目被强烈设想为使控制理论家更接近数据科学家，以便这两个研究团体可以一起工作，并回答重要的问题，例如：为什么大数据的价值传统上在控制中没有得到充分利用，控制理论可以从机器学习中获得哪些新的维度，反之亦然，以及需要哪些主要的分析和实验工具来使这种结合更成功。该研究还将支持博士和本科生的跨学科发展，以及机器学习在控制中的应用的研究生课程的开发。这项工作背后的主要技术理念将是利用大多数大规模动态网络在其动态中表现出大量冗余的事实。这些冗余可以由各种因素引起，如时间尺度分离、空间尺度分离、低秩可控性、光谱聚类、时间快照的相似性、控制目标的分离等。通过使用机器学习工具从状态和输入的在线测量中破译这些冗余，人们可以设计适当的分解规则，将网络划分为不重叠的组。然后可以使用无模型强化学习为每组独立学习多组复合控制器。相应地，网络的控制目标也将分解为局部（微观）和全局（宏观）的奖励函数。局部控制器采用保隐私群学习设计，全局控制器采用模型约简和滤波设计。该研究将以广域电力系统控制为例，使用来自相量测量单元（pmu）的流同步相量数据。验证实验将在北卡罗莱纳州立大学的网络物理系统测试台上进行。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

In the current state-of-the-art machine learning based real-time control of large complex networks such as electric power systems is largely bottlenecked by the curse of dimensionality. Even the simplest control designs demand numerical complexity to accomplish. The problem becomes even more challenging when the network model is unknown, due to which an additional learning time needs to be accommodated. This project will take a new stance for solving this problem, and develop a suite of hierarchical or nested machine learning-based schemes that take advantage of various forms of physical redundancies in the network dynamics to learn only the most important traits of its behavior instead of wasting time in learning minor traits that may improve the closed-loop performance only by a small amount. This selective learning approach will reduce learning time by several orders of magnitude, making real-time control more tractable and more implementable. Products will include numerical algorithms that are applicable across a wide range of machine learning based control. In terms of societal impact, the project is strongly envisioned to bring control theorists closer to data scientists so that these two research communities can work together, and answer important questions such as: why the value of big data has traditionally been under-utilized in controls, what new dimensions can control theory gain from machine learning and vice versa, and what primary analytical and experimental tools are needed to make this marriage more successful. The research will also support the cross-disciplinary development of a diverse cohort of PhD and undergraduate students, and the development of a graduate-level course on the applications of machine learning in control.The main technical philosophy behind this work will be to exploit the fact that most large-scale dynamic networks exhibit a lot of redundancy in their dynamics. These redundancies can arise from various factors such as time-scale separation, spatial-scale separation, low-rank controllability, spectral clustering, similarity in temporal snapshots, separation in the control objectives, etc. By deciphering these redundancies from online measurements of states and inputs using machine learning tools, one can devise appropriate decomposition rules to partition the network into non-overlapping groups. Multiple sets of composite controllers can then be learned independently for each group using model-free reinforcement learning. Accordingly, the control goals of the network will also be decomposed into local (microscopic) and global (macroscopic) reward functions. Local controllers will be designed via privacy preserving group learning, and the global controllers via model reduction and filtering. The study will be driven by examples from wide-area control of power systems using streaming Synchrophasor data from Phasor Measurements Units (PMUs). Validation experiments will be carried out in a cyber-physical systems testbed at North Carolina State University.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.