CPS: Small: Data-Driven Reinforcement Learning Control of Large CPS Networks using Multi-Stage Hierarchical Decompositions

在当前最先进的基于机器学习的电力系统等大型复杂网络的实时控制中，维度诅咒在很大程度上是制约实时控制的瓶颈。即使是最简单的控制设计也需要复杂的数值运算才能完成。当网络模型未知时，该问题变得更加具有挑战性，因此需要适应额外的学习时间。这个项目将采取一种新的立场来解决这个问题，并开发一套基于分层或嵌套的机器学习方案，该方案利用网络动力学中各种形式的物理冗余来仅学习其行为的最重要特征，而不是浪费时间学习可能仅对闭环系统性能有很小改善的次要特征。这种选择性学习方法将学习时间减少几个数量级，使实时控制更容易处理和更容易实现。产品将包括适用于广泛的基于机器学习的控制的数值算法。在社会影响方面，该项目旨在让控制理论家更接近数据科学家，以便这两个研究团体能够合作，并回答一些重要问题，例如：为什么大数据的价值在控制方面传统上没有得到充分利用，哪些新的维度可以控制理论从机器学习中获得的收益，反之亦然，以及需要哪些主要的分析和实验工具来使这一联姻更成功。这项研究还将支持一批不同的博士和本科生的跨学科发展，以及开发一门关于机器学习在控制中的应用的研究生课程。这项工作背后的主要技术哲学将是利用大多数大型动态网络在其动力学中表现出大量冗余的事实。这些冗余可能是由各种因素引起的，例如时间尺度分离、空间尺度分离、低阶可控性、谱聚类、时间快照的相似性、控制目标的分离等。通过使用机器学习工具从状态和输入的在线测量中解密这些冗余，可以设计适当的分解规则将网络划分为不重叠的组。然后，可以使用无模型强化学习为每个组独立地学习多组复合控制器。相应地，网络的控制目标也将分解为局部(微观)和全局(宏观)奖励函数。局部控制器通过保护隐私的群体学习来设计，全局控制器通过模型降阶和滤波来设计。这项研究将通过使用来自相量测量单元(PMU)的同步相量流数据的电力系统广域控制的例子来推动。验证实验将在北卡罗来纳州立大学的网络物理系统试验台上进行。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

In the current state-of-the-art machine learning based real-time control of large complex networks such as electric power systems is largely bottlenecked by the curse of dimensionality. Even the simplest control designs demand numerical complexity to accomplish. The problem becomes even more challenging when the network model is unknown, due to which an additional learning time needs to be accommodated. This project will take a new stance for solving this problem, and develop a suite of hierarchical or nested machine learning-based schemes that take advantage of various forms of physical redundancies in the network dynamics to learn only the most important traits of its behavior instead of wasting time in learning minor traits that may improve the closed-loop performance only by a small amount. This selective learning approach will reduce learning time by several orders of magnitude, making real-time control more tractable and more implementable. Products will include numerical algorithms that are applicable across a wide range of machine learning based control. In terms of societal impact, the project is strongly envisioned to bring control theorists closer to data scientists so that these two research communities can work together, and answer important questions such as: why the value of big data has traditionally been under-utilized in controls, what new dimensions can control theory gain from machine learning and vice versa, and what primary analytical and experimental tools are needed to make this marriage more successful. The research will also support the cross-disciplinary development of a diverse cohort of PhD and undergraduate students, and the development of a graduate-level course on the applications of machine learning in control.The main technical philosophy behind this work will be to exploit the fact that most large-scale dynamic networks exhibit a lot of redundancy in their dynamics. These redundancies can arise from various factors such as time-scale separation, spatial-scale separation, low-rank controllability, spectral clustering, similarity in temporal snapshots, separation in the control objectives, etc. By deciphering these redundancies from online measurements of states and inputs using machine learning tools, one can devise appropriate decomposition rules to partition the network into non-overlapping groups. Multiple sets of composite controllers can then be learned independently for each group using model-free reinforcement learning. Accordingly, the control goals of the network will also be decomposed into local (microscopic) and global (macroscopic) reward functions. Local controllers will be designed via privacy preserving group learning, and the global controllers via model reduction and filtering. The study will be driven by examples from wide-area control of power systems using streaming Synchrophasor data from Phasor Measurements Units (PMUs). Validation experiments will be carried out in a cyber-physical systems testbed at North Carolina State University.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.