Reinforcement Learning for Automated Flowsheet Synthesis of Steady-State Processes

流程合成是化工过程概念设计的关键步骤。就其本质而言，这是一个难以形式化的创造性过程。然而，目前计算机辅助流程图合成的方法大多是形式化的算法，这些算法采用基于知识的规则来创建流程图备选方案，并使用数学规划从较大的备选方案集（通常通过上层结构定义）中选择最佳流程图。该项目的目标是利用最近在强化学习（RL）方面取得的进展，实现稳态化学过程的自动化但具有创造性（这里：创造性和探索性）的流程图合成。RL环境是一个过程模拟器，它包含先验的物理知识，即物理化学性质数据和一组通用过程单元模型。RL代理可以一步一步地建立流程表，修改流程表，并在流程模拟器中对流程表进行评估，以获得反馈/奖励。代理没有化学工程的先验知识，并被训练成通过与模拟器的自动交互来创建流程图。核心工作假设是，这种设置能够自主产生可行的工艺流程图，这些流程流程图对于给定的成本函数是接近最优或最优的。在优先计划的第一阶段，汉堡集团（化学工程）将为几个示例问题开发和实施模拟环境。采用简化的过程单元快捷模型和代理模型来获得鲁棒的仿真环境。尽管降低了模型深度，但动作空间仍然很大（由于大量可想象的流程图变体）和参数化（由于过程单元的连续参数）。格林集团（机器学习）将开发量身定制的强化学习方法来应对这些挑战。分层强化学习和参数化动作空间的新方法将被研究和发展。为了改进前瞻性规划和探索，流程图综合问题将嵌入竞争性的双人游戏设置中，该设置已在联合初步工作中引入。这允许使用经典游戏应用程序（国际象棋，围棋）开发的算法进行有效的自我训练。在所有项目阶段都需要两个团队之间的密切合作，以找到双人游戏的最佳规则（允许的行动，目标和奖励）和最佳代理结构（特征选择，分层决策）。该项目位于优先计划合作矩阵的领域F，主要在研究领域#6创造力。与其他项目有很大的合作潜力，因为这个项目与所有项目共享共同利益，这些项目将开发健壮的模拟环境，创造性地设计过程单元或分子结构，和/或优化过程。

Flowsheet synthesis is a key step in conceptual design of chemical processes. By its nature, it is a creative process that is hard to formalize. Current methods of computer-aided flowsheet synthesis are however mostly formalized algorithms that employ knowledge-based rules for creating flowsheet alternatives and mathematical programming for selecting optimal flowsheets from larger sets of alternatives (typically defined via superstructures). The goal of this project is using the recently achieved progress in reinforcement learning (RL) for automated but creative (here: inventive and explorative) flowsheet synthesis of steady-state chemical processes. The RL environment is a process simulator that contains the a priori physical knowledge, i.e. physico-chemical property data and a set of general process unit models. Step by step, the RL agent can set up process flowsheets, modify them, and evaluate them in the process simulator to obtain feedback/reward. The agent has no prior knowledge of chemical engineering and is trained to create flowsheets solely through automated interaction with the simulator. The central work hypothesis is that this setup is able to autonomously produce feasible process flowsheets that are near-optimal or optimal regarding a given cost function.In the first period of the Priority Programme, the Burger group (Chemical Engineering) will develop and implement simulation environments for several example problems. Simplified shortcut and surrogate models for process units are used to obtain robust simulation environments. Despite the reduced model depth, the action space remains large (due to a large number of conceivable flowsheet variants) and parameterized (due to continuous parameters of the process units). The Grimm Group (Machine Learning) will develop tailored RL methods to cope with these challenges. Hierarchical RL and novel methods for parametrized action spaces will be investigated and developed. For improved forward planning and exploration, the flowsheet synthesis problem will be embedded into a competitive two-player game setup, which has been introduced in joint preliminary work. This allows for efficient training in self-play using algorithms developed for classical game applications (Chess, Go). Close collaboration between both groups is required in all project stages to find optimal rules of the two-player game (allowed actions, objectives and reward) and optimal agent structures (feature selection, hierarchical decisions).The project is located in field F of the Priority Programme’s collaboration matrix and primarily in the research area #6 creativity. Great collaboration potential with other projects is given, because this project shares common interests with all projects that will develop robust simulation environments, creatively design process units or molecular structures, and/or optimize processes.