FRR: Symmetric Policy Learning for Robotic Manipulation

机器学习最近对机器人技术产生了重大影响，使机器人能够以手动编程难以解决的方式学习解决问题。不幸的是，目前这种学习大多发生在模拟中，而不是在真实的世界中。这种方法是必要的，因为今天的机器学习算法通常需要大量的数据来学习任何有意义的东西，而模拟是将这些数据应用于其中的最直接方式。然而，模拟和真实的经验之间的不匹配带来了挑战，理想情况下，机器人能够像人类和动物一样从物理世界的试错经验中学习。通过简化和概括真实的经验，这些系统可以从有限的真实数据中获得更多。该项目探索了一种通过将问题对称性纳入机器学习来实现所需简化的方法。初步工作表明，这是一种很有前途的方法，我们希望能够显着提高机器人学习的效率。这项工作具有重要的潜力，可以影响各种领域的应用，包括国防应用，空间应用，仓储和物流应用，医疗保健应用以及家庭应用。这项工作的结果将在研究界和公众中广泛传播。今天机器人技术中使用的几乎所有规划和学习方法都依赖于世界模型--有时是错误的模型。理想情况下，机器人系统将有能力在线适应真实的世界的细微差别，因为他们遇到。这种适应的主要范例是强化学习（RL）和模仿学习（IL）。然而，今天的RL和IL算法的样本效率还不足以直接在真实的世界中学习。样本效率意味着算法可以从少量经验中彻底学习。如果样本效率提高到可以在物理机器人系统上有意义地进行RL的程度，那么它可以显着提高机器人控制策略的可靠性，特别是对于难以建模的问题，如接触丰富的操作。这是这个项目的重点-提高样本效率，使机器人可以通过RL直接在真实的世界中在线学习和适应，并通过IL从少量演示中学习。该项目将通过利用一类新的对称神经模型来实现这一目标，这些模型对许多机器人领域中存在的问题对称性进行编码。初步工作表明，这些模型在某些情况下可以加快学习速度。该项目的主要目标是：1）扩展现有的对称学习方法，以处理不完全对称的领域; 2）探索对象因子对称模型; 3）探索视觉力领域的对称学习;四、该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准。

Machine learning has recently had a major impact on robotics, enabling robots to learn to solve problems in ways that would have been difficult to program manually. Unfortunately, most of this learning currently happens in simulation, and not in the real world. This approach is necessary because today’s machine learning algorithms generally require large amounts of data to learn anything meaningful and simulation is the most direct way of bringing this data to bear. However, there are challenges that come from mismatches between simulation and real experience and, ideally, robots would be able to learn from trial-and-error experience in the physical world as humans and animals do. By simplifying and generalizing real experiences, these systems can get more out of the limited amount of real-world data. This project explores an approach to achieve needed simplifications by incorporating problem symmetries into machine learning. Preliminary work suggests that this is a promising approach and we expect to be able to significantly improve the efficacy of robotic learning in general. This work has significant potential to impact applications in a variety of fields including defense applications, space applications, warehousing and logistics applications, healthcare applications, and applications in the home. The results of this work will be disseminated widely both in the research community and to the public at large.Nearly all planning and learning methods used in robotics today depend on models of the world – models that are sometimes wrong. Ideally, robotic systems would have the ability to adapt online to the nuances of the real world as they are encountered. The dominant paradigms for this type of adaptation are reinforcement learning (RL) and imitation learning (IL). However, today’s RL and IL algorithms are not nearly sample efficient enough to learn directly in the real world. Sample efficiency means that the algorithm can learn thoroughly from a small number of experiences. If sample efficiency were improved to the point that one could meaningfully do RL on physical robotic systems, it could dramatically improve the reliability of robotic control policies, especially for hard-to-model problems like contact-rich manipulation. This is the focus of this project – to improve sample efficiency so that robots can learn and adapt online directly in the real world via RL and learn from a small number of demonstrations via IL. The project will achieve this goal by leveraging a new class of symmetric neural models that encode problem symmetries present in many robotics domains. Preliminary work suggests these models can speed up learning by orders of magnitude in some cases. The project has the following main aims: 1) to expand current symmetric learning methods to handle domains with imperfect symmetries; 2) to explore object factored symmetric models; 3) to explore symmetric learning in visual force domains; 4) to explore policy learning directly on physical robotic systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.