SBIR Phase I: Reinforcement Learning for Guidance and Control of Spacecraft

小型企业创新研究(SBIR)第一阶段项目的更广泛的影响/商业潜力是一种耦合的制导和控制飞行软件解决方案，使多航天器系统能够在其站位保持和自我分配方面真正自主。拟议的创新是让深度强化学习(DRL)代理学习如何自主确定和指挥动作，以产生所需的航天器编队。拟议的项目可以通过将人类从闭环控制系统中移除来降低客户的运营成本。拟议的创新可扩展到大量航天器的系统，而不会增加飞行操作的成本。对客户的其他潜在好处是降低风险：DRL代理不需要航天器子系统和轨道动力学的准确模型，并且可以实时学习，因此对非标称系统性能和意外扰动具有健壮性。该项目的好处可能包括DRL代理人为任务设计发现新的制导和控制解决方案，这些解决方案不是从传统的轨道动力学方法中得知的。潜在的更广泛的社会影响包括使深空门户运营和科学任务能够在大范围内对现场、同时测量进行采样，从而为地球轨道或深空的研究或商业应用提供宝贵的科学数据回报。这个小型企业创新研究第一阶段项目将展示实施DRL作为真正自主的航天器制导和控制解决方案的技术可行性。推动这个项目的挑战是多航天器系统的控制，由于相对运动的非线性方程，机动规划既不直观也不简单。在历史上，通过简化圆形轨道的假设和线性化的相对运动方程来找到解决方案。在这项研究计划中避免了这样的假设。主要研究目标是使用NASA通用任务分析工具(GMAT)的高保真模型培训一名DRL特工。首先，实现特定编队或分布的问题将被描述为马尔可夫决策过程。接下来，将使用TensorFlow、Python和GMAT开发软件基础设施。在这个框架内，DRL代理将接受培训，学习一项政策，即在可用推进剂和驱动限制等操作限制下，将航天器操纵到指定的编队中。预期的技术成果包括实现航天器任务协调的政策上和政策外方法的比较，基于DRL的制导和控制飞行软件的技术可行性的展示，以及基于DRL的控制的局限性的表征。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The broader impact/commercial potential of this Small Business Innovation Research (SBIR) Phase I project is a coupled guidance and control flight software solution that enables multi-spacecraft systems to be truly autonomous in their station-keeping and self-distribution. The proposed innovation is for a deep reinforcement learning (DRL) agent to learn how to autonomously determine and command maneuvers that yield a desired spacecraft formation. The proposed project could reduce customers’ cost of operations by removing humans from the closed loop control system. The proposed innovation scales to systems of large numbers of spacecraft without scaling the cost of flight operations. Other potential benefits to customers are risk reductions: a DRL agent does not require an exact model of spacecraft subsystems and orbital dynamics and can learn in real-time, thus being robust to off-nominal system performance and unexpected perturbations. Benefits of this project may include a DRL agent discovering novel guidance and control solutions for mission designs that are not known from legacy orbital dynamics approaches. Potential broader societal impacts include enabling Deep Space Gateway operations and science missions to sample in-situ, simultaneous measurements over a large area, resulting in valuable science data returns for research or commercial applications in Earth orbit or deep space.This Small Business Innovation Research Phase I project will demonstrate the technical feasibility of implementing DRL as a solution for truly autonomous spacecraft guidance and control. The challenge motivating this project is the control of multi-spacecraft systems, where maneuver planning is neither intuitive nor straightforward due to the nonlinear equations of relative motion. Historically, solutions are found by making simplifying assumptions of circular orbits and linearized equations of relative motion. Such assumptions are avoided in this research plan. The primary research objective is to train a DRL agent using the high-fidelity model of NASA’s General Mission Analysis Tool (GMAT). First, the problem of achieving a particular formation or distribution will be formulated as a Markov Decision Process. Next, software infrastructure will be developed using TensorFlow, Python, and GMAT. Within this framework, the DRL agent will be trained to learn a policy that maneuvers the spacecraft into a specified formation, subject to operations constraints like available propellant and actuation limits. Anticipated technical results include comparisons of on-policy versus off-policy approaches to achieving coordinated spacecraft mission, demonstration of the technical feasibility of DRL-based flight software for guidance and control, and a characterization of the limitations of DRL-based control.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.