CPS: Small: Data-Driven Reinforcement Learning Control of Large CPS Networks using Multi-Stage Hierarchical Decompositions

CPS：小型：使用多级分层分解对大型 CPS 网络进行数据驱动的强化学习控制

• 批准号: 1931932
• 负责人: Wenyuan Tang
• 金额: $ 35.32万
• 依托单位: North Carolina State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-01-01 至 2024-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1931932&HistoricalAwards=false
• 关键词: CPS; Small; Data; Driven; Reinforcement

In the current state-of-the-art machine learning based real-time control of large complex networks such as electric power systems is largely bottlenecked by the curse of dimensionality. Even the simplest control designs demand numerical complexity to accomplish. The problem becomes even more challenging when the network model is unknown, due to which an additional learning time needs to be accommodated. This project will take a new stance for solving this problem, and develop a suite of hierarchical or nested machine learning-based schemes that take advantage of various forms of physical redundancies in the network dynamics to learn only the most important traits of its behavior instead of wasting time in learning minor traits that may improve the closed-loop performance only by a small amount. This selective learning approach will reduce learning time by several orders of magnitude, making real-time control more tractable and more implementable. Products will include numerical algorithms that are applicable across a wide range of machine learning based control. In terms of societal impact, the project is strongly envisioned to bring control theorists closer to data scientists so that these two research communities can work together, and answer important questions such as: why the value of big data has traditionally been under-utilized in controls, what new dimensions can control theory gain from machine learning and vice versa, and what primary analytical and experimental tools are needed to make this marriage more successful. The research will also support the cross-disciplinary development of a diverse cohort of PhD and undergraduate students, and the development of a graduate-level course on the applications of machine learning in control.The main technical philosophy behind this work will be to exploit the fact that most large-scale dynamic networks exhibit a lot of redundancy in their dynamics. These redundancies can arise from various factors such as time-scale separation, spatial-scale separation, low-rank controllability, spectral clustering, similarity in temporal snapshots, separation in the control objectives, etc. By deciphering these redundancies from online measurements of states and inputs using machine learning tools, one can devise appropriate decomposition rules to partition the network into non-overlapping groups. Multiple sets of composite controllers can then be learned independently for each group using model-free reinforcement learning. Accordingly, the control goals of the network will also be decomposed into local (microscopic) and global (macroscopic) reward functions. Local controllers will be designed via privacy preserving group learning, and the global controllers via model reduction and filtering. The study will be driven by examples from wide-area control of power systems using streaming Synchrophasor data from Phasor Measurements Units (PMUs). Validation experiments will be carried out in a cyber-physical systems testbed at North Carolina State University.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在当前最先进的基于机器学习的大型复杂网络（如电力系统）的实时控制中，很大程度上受到维数灾难的影响。即使是最简单的控制设计也需要复杂的数值来实现。当网络模型未知时，问题变得更具挑战性，因此需要适应额外的学习时间。该项目将采取一种新的立场来解决这个问题，并开发一套基于分层或嵌套机器学习的方案，利用网络动态中各种形式的物理冗余来学习其行为的最重要特征，而不是浪费时间学习可能只会少量提高闭环性能的次要特征。这种选择性学习方法将减少几个数量级的学习时间，使实时控制更易于处理和实现。产品将包括适用于各种基于机器学习的控制的数值算法。在社会影响方面，该项目的强烈设想是让控制理论家更接近数据科学家，以便这两个研究社区可以共同工作，并回答重要的问题，例如：为什么大数据的价值在控制中传统上没有得到充分利用，控制理论可以从机器学习中获得什么新的维度，反之亦然，以及需要哪些主要的分析和实验工具来使这种婚姻更加成功。该研究还将支持博士生和本科生的跨学科发展，以及开发关于机器学习在控制中应用的研究生课程。这项工作背后的主要技术理念将是利用大多数大规模动态网络在其动态中表现出大量冗余的事实。这些冗余可能会出现从各种因素，如时间尺度分离，空间尺度分离，低秩可控性，谱聚类，时间快照的相似性，分离的控制目标等，通过破译这些冗余的状态和输入的在线测量使用机器学习工具，可以设计适当的分解规则，将网络划分为不重叠的组。然后，可以使用无模型强化学习为每个组独立地学习多组复合控制器。相应地，网络的控制目标也将被分解为局部（微观）和全局（宏观）奖励函数。局部控制器将通过隐私保护组学习设计，全局控制器通过模型简化和过滤。该研究将驱动的例子，从广域控制的电力系统使用流同步相量数据相量测量单元（PMU）。验证实验将在北卡罗来纳州州立大学的网络物理系统测试平台上进行。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估来支持。