RI: Medium: Learning MDP abstractions for Autonomous Systems using Variational Methods and Symmetry Groups

自动驾驶汽车、医院平台和家用机器人等自主系统具有巨大的潜在社会和经济效益，有能力改变未来的工作、医疗保健和我们的日常生活。然而，成功的自主要求机器人能够自己做决定，并从自己的经验中学习。这可能具有挑战性，因为现实世界丰富而复杂，自主机器人系统可能会被细节所迷惑。有时候，一个自治系统不能正确地泛化：它会认为两个非常相似的情况根本不同。该项目旨在开发新的方法来学习任务和领域适当的抽象，这将有助于自治系统更有效地泛化到新的情况。更好的抽象将允许自治系统更有效地做出决策，从而改进学习和有效控制。该项目将研究马尔可夫决策过程和强化学习的决策理论框架中的抽象问题，这些框架已被广泛用作自动化决策的框架。强化学习的最新进展使自主代理和机器人能够完成具有挑战性的任务，有时甚至超过人类专家。然而，这在样本和计算复杂性方面的成本都非常高；数以百万计的训练步骤和数天的训练时间是典型的，即使是在类似游戏的环境中。该项目将开发使这一过程更有效的方法，通过显式地将学习良好抽象的目标编码到代理的成本函数中。具体来说，pi将研究和开发将大型连续决策问题压缩为小型离散决策问题的方法，以及将编码问题中不相关的显式对称约束纳入其中的方法。这些方法将在各种不同复杂性的领域中进行评估，包括涉及移动导航和机器人操作的自主系统任务。总体目标是开发提高学习效率、抽象质量和对新任务和新情况的泛化的方法。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Autonomous systems such as self-driving vehicles, hospital platforms, and household robots have great potential social and economic benefits, with the ability to transform the future of work, healthcare, and our daily routines. However, successful autonomy requires the robot to be able to make its own decisions and learn from its own experiences. This can be challenging because the real world is rich and complex and autonomous robotic systems can become confused by the details. It is sometimes the case that an autonomous system will not generalize properly: it will perceive two very similar situations to be fundamentally different. This project aims to develop new methods for learning task- and domain-appropriate abstractions that will help autonomous systems generalize to new situations more effectively. Better abstraction will allow autonomous systems to make decisions more efficiently leading to improved learning and effective control.This project will study the problem of abstraction within the decision-theoretic framework of Markov decision processes and reinforcement learning, which have been widely used as a framework for automated decision making. Recent advances in reinforcement learning have enabled autonomous agents and robots to accomplish challenging tasks, sometimes even surpassing human experts. However, this comes at an extremely high cost, both in sample and computational complexity; millions of training steps and days of training time are typical, even in game-like environments. This project will develop approaches for making this process much more efficient, by explicitly encoding objectives for learning good abstractions into the agent's cost function. Specifically, the PIs will study and develop approaches for compressing large continuous decision-making problems into small discrete ones, as well as approaches that incorporate explicit symmetry constraints that encode irrelevances in the problem. These methods will be evaluated on a variety of domains of varying complexity, including tasks on autonomous systems involving mobile navigation and robot manipulation. The overall objective is to develop approaches that improve learning efficiency, abstraction quality, and generalization to new tasks and situations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.