SBIR Phase I: Reinforcement Learning for Guidance and Control of Spacecraft

这个小型企业创新研究（SBIR）第一阶段项目的更广泛的影响/商业潜力是一个耦合的制导和控制飞行软件解决方案，使多航天器系统能够真正自主地保持位置和自我分配。提出的创新是深度强化学习（DRL）代理学习如何自主确定和命令机动，产生所需的航天器编队。拟议的项目可以通过将人类从闭环控制系统中去除来降低客户的运营成本。拟议的创新可扩展到大量航天器的系统，而不会扩展飞行操作的成本。对客户的其他潜在好处是降低风险：DRL代理不需要航天器子系统和轨道动力学的精确模型，并且可以实时学习，因此对偏离标称的系统性能和意外扰动具有鲁棒性。该项目的好处可能包括DRL代理发现新的制导和控制解决方案的使命设计，是不知道从传统的轨道动力学方法。潜在的更广泛的社会影响包括使深空网关操作和科学任务能够在大面积上进行现场采样，同时测量，从而为地球轨道或深空的研究或商业应用提供有价值的科学数据回报。激励这个项目的挑战是多航天器系统的控制，其中机动规划既不直观也不简单，由于相对运动的非线性方程。历史上，通过简化圆形轨道的假设和相对运动的线性方程来找到解决方案。在本研究计划中避免了这种假设。主要的研究目标是训练一个DRL代理使用高保真模型的美国宇航局的通用使命分析工具（GMAT）。首先，实现特定的形成或分布的问题将被公式化为马尔可夫决策过程。接下来，将使用TensorFlow、Python和GMAT开发软件基础设施。在此框架内，DRL代理将接受培训，以学习将航天器机动到指定编队的策略，并受到可用推进剂和驱动限制等操作限制。预期的技术成果包括：比较实现协调的航天器使命的政策内和政策外方法，演示用于制导和控制的基于DRL的飞行软件的技术可行性，以及日间行车灯局限性的表征-该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The broader impact/commercial potential of this Small Business Innovation Research (SBIR) Phase I project is a coupled guidance and control flight software solution that enables multi-spacecraft systems to be truly autonomous in their station-keeping and self-distribution. The proposed innovation is for a deep reinforcement learning (DRL) agent to learn how to autonomously determine and command maneuvers that yield a desired spacecraft formation. The proposed project could reduce customers’ cost of operations by removing humans from the closed loop control system. The proposed innovation scales to systems of large numbers of spacecraft without scaling the cost of flight operations. Other potential benefits to customers are risk reductions: a DRL agent does not require an exact model of spacecraft subsystems and orbital dynamics and can learn in real-time, thus being robust to off-nominal system performance and unexpected perturbations. Benefits of this project may include a DRL agent discovering novel guidance and control solutions for mission designs that are not known from legacy orbital dynamics approaches. Potential broader societal impacts include enabling Deep Space Gateway operations and science missions to sample in-situ, simultaneous measurements over a large area, resulting in valuable science data returns for research or commercial applications in Earth orbit or deep space.This Small Business Innovation Research Phase I project will demonstrate the technical feasibility of implementing DRL as a solution for truly autonomous spacecraft guidance and control. The challenge motivating this project is the control of multi-spacecraft systems, where maneuver planning is neither intuitive nor straightforward due to the nonlinear equations of relative motion. Historically, solutions are found by making simplifying assumptions of circular orbits and linearized equations of relative motion. Such assumptions are avoided in this research plan. The primary research objective is to train a DRL agent using the high-fidelity model of NASA’s General Mission Analysis Tool (GMAT). First, the problem of achieving a particular formation or distribution will be formulated as a Markov Decision Process. Next, software infrastructure will be developed using TensorFlow, Python, and GMAT. Within this framework, the DRL agent will be trained to learn a policy that maneuvers the spacecraft into a specified formation, subject to operations constraints like available propellant and actuation limits. Anticipated technical results include comparisons of on-policy versus off-policy approaches to achieving coordinated spacecraft mission, demonstration of the technical feasibility of DRL-based flight software for guidance and control, and a characterization of the limitations of DRL-based control.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.