Fully Bayesian Reinforcement Learning for Control of Continuous Industrial Processes

这一令人兴奋和创新的PHD与NSG合作，涉及监测和控制连续制造过程的设置，重点是保证产品质量和将这样做的成本降至最低，例如，通过最小化用于保证产品满足特定规格(如厚度或缺陷率)的多余材料的数量。重点是玻璃的制造和处理。在这样的设置中，在控制输入改变和可观察到的响应之间通常有很长的等待时间(即几分钟)。在这些环境中应用反馈控制是具有挑战性的，因此现有的工程解决方案通常使用过程的物理模型，并采用基于预测模型的控制。虽然这确实使得在产品中产生所需的变化成为可能，但该方法依赖于工艺的物理模型和已知的传感器模型。这些模型一般都很好理解，但在某些方面，不可能建立准确的模型，例如，可以推断在某个历史时期，加热元件上施加的功率的变化如何影响厚度剖面的精细细节。此外，随着时间的推移，实际情况会发生变化(例如，由于阀门磨损或最近没有进行定期维护)，虽然可以开发变通方法来适应这些变化，但这些变通方法可能会失败。这种故障可能会导致产品质量的突然和显著下降。因此，根本的挑战是开发一种充分利用离线历史数据的控制策略；捕捉对过程和传感器性能的广泛但不完整理解的参数化模型；从这些模型获得的离线模拟体验；来自传感器的在线数据。开发这样的控制策略将需要数值贝叶斯推理算法(例如马尔科夫链蒙特卡罗)以利用历史数据和领域专家的现有理解的方式对模型进行推理。借鉴最近强化学习(RL)在其他领域的成功应用，强化学习将被用来学习如何在给定的推断模型下最好地应用控制。这种RL是计算密集型的，因此需要使用高性能计算资源。

This exciting and innovative PhD, in partnership with NSG, relates to settings where a continuous manufacturing process is monitored and so controlled with a focus on both guaranteeing the quality of the product and minimising the costs of doing so, e.g. by minimising the amount of excess material used to guarantee that certain specifications of the product (e.g. thickness or defect rate) are met. The focus is on manufacture and treatment of glass. In such settings there is often a significant latency (i.e. minutes) between the control input changing and the response being observable. It is challenging to apply feedback control in these contexts, so existing Engineering solutions often make use of physical models for the process and employ predictive model-based control. While this does make it possible to produce desired variations in the product, the approach relies on the physical models for the process and the models for the sensors to be known. These models are well understood in general, but there are aspects where it is not possible to build accurate models that, for example, can infer how the fine detail of the thickness profile is impacted by variation in the power applied to heating elements at some historic time. Furthermore, the real-world changes over time (e.g. because valves become worn or because scheduled maintenance has not occurred recently) and while it is possible to develop work-arounds to adapt to these changes, these work-arounds can fail. Such failures can result in sudden and significant degradation in the quality of product. The fundamental challenge is then to develop a control strategy that fully capitalises on: offline historic data; parameterised models that capture the extensive but incomplete understanding of the processes and sensors' performance; offline simulated experience derived from those models; online data from sensors. Developing such a control strategy will require numerical Bayesian inference algorithms (e.g. Markov Chain Monte Carlo) to make inferences about the models in a way that exploits the historic data and domain experts' existing understanding. Borrowing from recent successful applications of Reinforcement Learning (RL) in other domains, RL will then be used to learn how best to apply the control given the inferred model. Such RL is computationally intensive and will therefore require use of High-Performance Computing resources.