Reinforcement Learning and Kullback-Leibler Stochastic Optimal Control for Complex Networks

自然和人造的网络系统就在我们周围。电网和互联网是两个明显复杂的互联系统的例子，其中数百万的“代理人”渴望以能源或带宽的形式提取价值。虽然这些系统在用图论术语测量时是复杂的，但通信和能源系统的行为对最终用户（在世界上大多数地方）来说似乎是简单且高度可预测的。这一成功部分归功于管理系统范围供需平衡的分布式控制回路。互联网中分布式控制的一个例子是TCP/IP，以及大多数电网中的自动发电控制（AGC）。虽然分布式控制协议在通信应用中得到了高度发展和广泛接受，但在其他网络化系统（如电力和天然气分配）中却不那么真实。该项目旨在推进复杂互联系统的控制理论。 应用的重点是电力系统，但控制技术是通用的，可能有更广泛的影响。最近的控制创新在该项目中被强调为构建控制算法的基石，基于本地决策和整体管理的组合：1。 局部决策的控制技术将是该项目的一个主题，使用PI小组介绍的新Kullback-Leibler-Quadratic最优控制方法。 2.强化学习（RL）是Google最近成功的计算机游戏背后的引擎，也是在不确定的复杂环境中进行控制合成的自然框架。 PI及其同事最近提出的Zap Q学习算法是一类新的RL算法，它几乎是普遍稳定的，并且具有可证明的最优收敛速度。 3.平均场模型在电力系统中有着悠久的历史（起源于统计物理学），它们将被用来近似聚合行为，并作为构建控制聚合算法的基础。 算法设计将与仿真研究相结合，最初侧重于电力系统的应用。 智能电网技术的课程将得到加强，该项目将包括本科生的参与。该奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

Natural and man-made networked systems are all around us. The power grid and the Internet are two examples of apparently complex interconnected systems, in which millions of "agents" are eager to extract value in the form of energy or bandwidth. While these systems are complex when measured in graph-theoretic terms, the behavior of communication and energy systems appears simple and highly predictable to the end users (in most of the world). This success is due in part to distributed control loops that manage system-wide supply-demand balance. An example of distributed control in the Internet is TCP/IP, and automatic generation control (AGC) in most electric power grids. While distributed control protocols are highly developed and widely accepted in communication applications, this is less true in other networked systems such as electric power and natural gas distribution. This project aims to advance control theory for complex interconnected systems. The application focus is on power systems, but the control techniques are general and are likely to have far broader impact. Recent control innovations are highlighted in the project as building blocks in the construction of algorithms for control, based on a combination of local decision making and global management of the ensemble: 1. Control techniques for local decision making will be a theme of the project using a new Kullback-Leibler-Quadratic optimal control approach introduced by the PI's group. 2. Reinforcement learning (RL) is the engine behind Google's recent computer game successes and is a natural framework for control synthesis in an uncertain complex environment. The Zap Q-learning algorithms introduced recently by the PI and his colleagues are a new class of RL algorithms that are virtually universally stable and have provably optimal convergence rate. 3. Mean field models have a long history in power systems (with roots in statistical physics), they will be used to approximate aggregate behavior, and as a foundation to construct algorithms to control the aggregate. Algorithm design will be complemented with simulation studies, focusing initially on applications to power systems. A course in smart grid technologies will be augmented and the project will include participation from undergraduate students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.