Deep reinforcement learning for the real time control of stormwater systems
Deep reinforcement learning for the real time control of stormwater systems
复制标题
DOI：
10.1016/j.advwatres.2020.103600
复制
发表时间：
2020-06
影响因子：
4.7
通讯作者：
Abhiram Mullapudi;Matthew J. Lewis;C. Gruden;B. Kerkez
中科院分区：
文献类型：
--
作者：
Abhiram Mullapudi;Matthew J. Lewis;C. Gruden;B. Kerkez
新一代智能雨水系统有望通过实时控制提高现有基础设施的性能，从而减少对新建筑的需求。智能雨水系统通过控制分布式资产（如阀门、闸门和泵）来动态调整其对单个风暴的响应。本文介绍了一种基于强化学习（RL）的实时控制方法，该方法已成为人工智能界自主控制的最新方法。使用深度神经网络，基于RL的控制器通过与其控制的系统交互来学习控制策略-有效地尝试各种控制策略，直到收敛到实现期望目标的控制策略。本文提出并实现了一种城市雨水系统实时控制的RL算法。该算法训练RL代理在数千个模拟风暴场景中控制分布式雨水系统中的阀门，以实现系统中的水位和流量设定点。该算法首先评估一个单独的雨水流域的控制，之后，它是适应多个流域在一个较大的流域（4平方公里）的控制。结果表明，RL可以非常有效地控制个别网站。性能是高度敏感的RL代理的奖励制定。一般来说，更明确的指导导致更好的控制性能，更快速和稳定的学习过程的收敛。虽然多个分布式站点的控制也显示出减少洪水和峰值流量的希望，但控制大型系统的复杂性带来了一些警告。RL控制器的性能对深度神经网络的公式化非常敏感，并且需要大量的计算资源来实现合理的性能增强。总体而言，受控系统的性能明显优于非受控系统，特别是在高强度和持续时间的风暴中。一个坦率的讨论，这应该允许RL的优点和缺点时，实施它的实时控制雨水系统的考虑。还提供了完整的仿真环境和控制算法的开源实现。
A new generation of smart stormwater systems promises to reduce the need for new construction by enhancing the performance of the existing infrastructure through real-time control. Smart stormwater systems dynamically adapt their response to individual storms by controlling distributed assets, such as valves, gates, and pumps. This paper introduces a real-time control approach based on Reinforcement Learning (RL), which has emerged as a state-of-the-art methodology for autonomous control in the artificial intelligence community. Using a Deep Neural Network, a RL-based controller learns a control strategy by interacting with the system it controls - effectively trying various control strategies until converging on those that achieve a desired objective. This paper formulates and implements a RL algorithm for the real-time control of urban stormwater systems. This algorithm trains a RL agent to control valves in a distributed stormwater system across thousands of simulated storm scenarios, seeking to achieve water level and flow set-points in the system. The algorithm is first evaluated for the control of an individual stormwater basin, after which it is adapted to the control of multiple basins in a larger watershed (4km2). The results indicate that RL can very effectively control individual sites. Performance is highly sensitive to the reward formulation of the RL agent. Generally, more explicit guidance led to better control performance, and more rapid and stable convergence of the learning process. While the control of multiple distributed sites also shows promise in reducing flooding and peak flows, the complexity of controlling larger systems comes with a number of caveats. The RL controller’s performance is very sensitive to the formulation of the Deep Neural Network and requires a significant amount of computational resource to achieve a reasonable performance enhancement. Overall, the controlled system significantly outperforms the uncontrolled system, especially across storms of high intensity and duration. A frank discussion is provided, which should allow the benefits and drawbacks of RL to be considered when implementing it for the real-time control of stormwater systems. An open source implementation of the full simulation environment and control algorithms is also provided.