Multi-level Reinforcement Learning for flow control

流量控制是对流体流场进行有针对性的操纵以实现规定的目标(例如，减少阻力)的过程。流量控制使用来自流量的信息(由传感器提供)来适应传入的扰动，并根据不断变化的流量条件进行调整。一般流量控制是一个在很大程度上没有解决的数学问题，出现在许多行业，包括汽车，航空航天和环境地下流动问题。将流量控制转化为实用工具的缺失因素是通用流量控制算法的发展，这些算法可以处理以下问题：(A)系统扰动中的不确定性(例如，扰动的速度和方向)，(B)流动模型参数中的不确定性，(C)观测数据的稀疏性(即，部分观测和噪声观测)(D)由于离散化和参数升级而导致的建模误差。在本方案中，强化学习(RL)算法将被用于通过可靠的模拟流动环境来学习一般流量控制策略。从应用的角度来看，开发的数学技术解决了两个应用中的流动控制：(A)通过对通过障碍物的不可压缩的Navier-Stokes流动进行流动控制来提高运输卡车的能源效率；(B)通过达西型地下流动的流动控制，在深部地质地层中安全有效地储存人为二氧化碳(CO2)。对于第一个应用，道路货运约占英国碳足迹的5%，通过流量控制来减少空气动力学阻力可以显著提高燃料效率，例如，阻力减少15%相当于节省约5%的燃料。对于二氧化碳储存应用，人类活动产生的二氧化碳，例如发电站或能源密集型行业产生的二氧化碳，可以注入深层盐碱地含水层，作为一种可能的缓解策略，以减少向大气中的人为二氧化碳排放。考虑到地下性质的固有不确定性，控制地下储存点的注入策略将最大限度地减少泄漏风险，同时最大限度地提高储存容量。

Flow control is the process of targeted manipulation of fluid flow fields to accomplish a prescribed objective (e.g. reduce drag). Flow control uses information from the flow (provided by sensors) to adapt to incoming perturbations and adjust to changing flow conditions. General flow control is a largely unsolved mathematical problem appearing in many industries, including automotive, aerospace and environmental subsurface flow problems. The missing ingredient for turning flow control into a practical tool is the development of general flow control algorithms that can handle the following: (a) uncertainties in the system perturbations (e.g. the speed and direction of the perturbation), (b) uncertainties in the flow model parameters, (c) sparsity of the observations (i.e. partial and noisy observations) (d) modelling errors due to discretization and parameter upscaling.In this proposal, Reinforcement Learning (RL) algorithms will be utilized to learn general flow control polices using reliable simulated flow environments. From an application point of view, the developed mathematical techniques address flow control in two applications: (a) increasing energy efficiency in transportation trucks by flow control of incompressible Navier-Stokes flow past an obstacle and (b) safe and efficient storage of anthropogenic carbon dioxide (CO2) in deep geological formations using flow control in a Darcy-type subsurface flow. For the first application, road freight transportation accounts for approximately 5% of the UK's carbon footprint and flow control to reduce the aerodynamic drag could significantly improve the fuel efficiency, for example a 15% reduction in drag is equivalent to about 5% in fuel savings. For the CO2 storage application, the produced CO2 by human activities, for example from a power stations or an energy-intensive industries, could be injected into deep saline aquifers as a possible mitigation strategy to reduce anthropogenic emissions of carbon dioxide into the atmosphere. The control of injection strategies in the subsurface storage sites, given the inherent uncertainties in the subsurface properties, would minimize the risk of leakage while maximising the storage capacity.