Hierarchical Reinforcement Learning Framework for Stochastic Spaceflight Campaign Design
Hierarchical Reinforcement Learning Framework for Stochastic Spaceflight Campaign Design
复制标题
DOI：
10.2514/1.a35122
复制
发表时间：
2021-03
期刊：
影响因子：
--
通讯作者：
Yuji Takubo;Hao Chen;K. Ho
中科院分区：
文献类型：
--
作者：
Yuji Takubo;Hao Chen;K. Ho
本文提出了一种分层强化学习体系结构，用于不确定条件下的多任务航天活动设计，包括飞行器设计、基础设施部署规划和航天运输调度。这个问题涉及到一个高维的设计空间，是具有挑战性的，特别是与不确定性。为了应对这一挑战，所开发的框架具有层次结构，具有强化学习和基于网络的混合整数线性规划（MILP），其中前者优化决策层决策（例如，在整个战役中使用的交通工具的设计，在战役中分配给每个使命的目的地需求），而后者优化了详细的任务级决策（例如，什么时候发射什么，从哪里发射到哪里）。该框架适用于一组人类月球探测活动的情况下，不确定的原位资源利用性能作为案例研究。这项工作的主要价值是它的集成快速增长的强化学习研究和现有的MILP为基础的空间物流方法，通过一个层次化的框架，以处理空间使命设计的不确定性，否则棘手的复杂性。这一独特的框架有望成为新兴的空间使命设计人工智能研究方向的关键垫脚石。
This paper develops a hierarchical reinforcement learning architecture for multimission spaceflight campaign design under uncertainty, including vehicle design, infrastructure deployment planning, and space transportation scheduling. This problem involves a high-dimensional design space and is challenging especially with uncertainty present. To tackle this challenge, the developed framework has a hierarchical structure with reinforcement learning and network-based mixed-integer linear programming (MILP), where the former optimizes campaign-level decisions (e.g., design of the vehicle used throughout the campaign, destination demand assigned to each mission in the campaign), whereas the latter optimizes the detailed mission-level decisions (e.g., when to launch what from where to where). The framework is applied to a set of human lunar exploration campaign scenarios with uncertain in situ resource utilization performance as a case study. The main value of this work is its integration of the rapidly growing reinforcement learning research and the existing MILP-based space logistics methods through a hierarchical framework to handle the otherwise intractable complexity of space mission design under uncertainty. This unique framework is expected to be a critical steppingstone for the emerging research direction of artificial intelligence for space mission design.