FRR: Symmetric Policy Learning for Robotic Manipulation

机器学习最近对机器人学产生了重大影响，使机器人能够学习以手动编程困难的方式解决问题。不幸的是，目前大多数这种学习都是在模拟中进行的，而不是在现实世界中。这种方法是必要的，因为今天的机器学习算法通常需要大量数据来学习任何有意义的东西，而模拟是利用这些数据的最直接方式。然而，模拟和真实体验之间的不匹配带来了挑战，理想情况下，机器人应该能够像人类和动物一样，从物理世界中的反复试验中学习经验。通过简化和概括真实体验，这些系统可以从有限的真实世界数据中获得更多。这个项目探索了一种通过将问题对称性纳入机器学习来实现所需简化的方法。初步工作表明，这是一种很有前途的方法，我们预计能够显着提高机器人学习的效率。这项工作具有巨大的潜力来影响各种领域的应用，包括国防应用、空间应用、仓储和物流应用、医疗保健应用以及家庭应用。这项工作的结果将在研究界和广大公众中广泛传播。几乎所有当今机器人学中使用的规划和学习方法都依赖于世界模型--这些模型有时是错误的。理想情况下，机器人系统将具有在线适应现实世界中遇到的细微差别的能力。这种适应的主要范式是强化学习(RL)和模仿学习(IL)。然而，今天的RL和IL算法的样本效率远远不足以在现实世界中直接学习。样本效率意味着算法可以从少量经验中彻底学习。如果样本效率提高到可以在物理机器人系统上进行有意义的RL的程度，就可以极大地提高机器人控制策略的可靠性，特别是对于像接触丰富的操作这样难以建模的问题。这就是这个项目的重点--提高样本效率，让机器人可以通过RL直接在现实世界中在线学习和适应，并通过IL从少量演示中学习。该项目将通过利用一类新的对称神经模型来实现这一目标，这些模型编码了许多机器人领域中存在的问题对称性。初步工作表明，在某些情况下，这些模型可以加快学习速度。该项目有以下主要目标：1)扩展当前的对称学习方法，以处理具有不完美对称性的领域；2)探索对象因数对称模型；3)探索视觉力域中的对称学习；4)直接在物理机器人系统上探索政策学习。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Machine learning has recently had a major impact on robotics, enabling robots to learn to solve problems in ways that would have been difficult to program manually. Unfortunately, most of this learning currently happens in simulation, and not in the real world. This approach is necessary because today’s machine learning algorithms generally require large amounts of data to learn anything meaningful and simulation is the most direct way of bringing this data to bear. However, there are challenges that come from mismatches between simulation and real experience and, ideally, robots would be able to learn from trial-and-error experience in the physical world as humans and animals do. By simplifying and generalizing real experiences, these systems can get more out of the limited amount of real-world data. This project explores an approach to achieve needed simplifications by incorporating problem symmetries into machine learning. Preliminary work suggests that this is a promising approach and we expect to be able to significantly improve the efficacy of robotic learning in general. This work has significant potential to impact applications in a variety of fields including defense applications, space applications, warehousing and logistics applications, healthcare applications, and applications in the home. The results of this work will be disseminated widely both in the research community and to the public at large.Nearly all planning and learning methods used in robotics today depend on models of the world – models that are sometimes wrong. Ideally, robotic systems would have the ability to adapt online to the nuances of the real world as they are encountered. The dominant paradigms for this type of adaptation are reinforcement learning (RL) and imitation learning (IL). However, today’s RL and IL algorithms are not nearly sample efficient enough to learn directly in the real world. Sample efficiency means that the algorithm can learn thoroughly from a small number of experiences. If sample efficiency were improved to the point that one could meaningfully do RL on physical robotic systems, it could dramatically improve the reliability of robotic control policies, especially for hard-to-model problems like contact-rich manipulation. This is the focus of this project – to improve sample efficiency so that robots can learn and adapt online directly in the real world via RL and learn from a small number of demonstrations via IL. The project will achieve this goal by leveraging a new class of symmetric neural models that encode problem symmetries present in many robotics domains. Preliminary work suggests these models can speed up learning by orders of magnitude in some cases. The project has the following main aims: 1) to expand current symmetric learning methods to handle domains with imperfect symmetries; 2) to explore object factored symmetric models; 3) to explore symmetric learning in visual force domains; 4) to explore policy learning directly on physical robotic systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.