Redesigning the ways we learn and express policies for autonomous agents, to increasing robustness, resolve simulation-to-real transfer, and enable sh

这个项目属于EPSRC机器人和人工智能技术研究领域(工程学主题)。当我开始我的博士之旅时，有两个广泛的想法，我相信这两个想法是相互联系的，我正在努力为之做出贡献并进一步发展。首先是发展分层强化学习的方法。强化学习(RL)是一种机器学习/人工智能方法，自20世纪中叶以来一直在研究，但直到最近四年才取得显著成功，可能最引人注目的是在2016年，一名基于强化学习的代理在一场全球报道的锦标赛上击败了世界围棋冠军李世石。在牛津机器人研究所，我们对将RL应用于物理机器人控制非常感兴趣，并在尚未看到真正有效成功的领域。RL的一个局限性是，它的训练速度非常慢，数据密集型，并且导致非常脆弱的控制策略，这些策略往往在他们已经学会解决的确切问题之外的任何方面都表现不佳。这些限制阻碍了这种令人兴奋的方法在激动人心的机器人学领域的应用，我的目标是通过构建一种执行分层RL的方法来帮助解决这个问题。这将为RL引入一种新的结构，其中将发生策略和学习，以及多个时间尺度或级别的“时间抽象”。这将借鉴更成熟和成功的控制理论领域的想法，其中分层控制早已被应用，但将这些想法带入我们新的、基于学习的范例中。第二，我希望通过学习在不同的时间抽象水平上控制，我将能够设计出一种以机器人为中心的方法来编码优化计算的机器人运动，这样它们可以自动排列并在不同的场景和不同的机器人中重复使用。我的一位主管Ioannis Havoutis参与了一个名为Memmo(运动记忆)的欧洲项目，该项目正在寻求一种方法，以先前存在的经过优化计算并保存(在运动记忆中)的运动序列为基础，为有手臂和腿的复杂机器人执行运动生成。至关重要的是，所寻求的方法应该允许这些序列是为不同的机器人平台设计的或在不同的机器人平台上执行的，然后使用Memmo系统进行运动。这需要解决以某种方式对来自运动的有用信息进行编码的问题，以便它可以被另一个系统利用。当前的方法没有充分考虑如何根据预期的应用最好地做到这一点，而是利用诸如自动编码或其他降维方法之类的传统方法来对这些运动进行编码。当涉及到在新的情况下借鉴、改变和重新使用这些动议时，这不会给我们带来更多的优势。相反，这些方法只是数据压缩。相比之下，我希望通过开发一种成熟的方法来解决这个问题，这种方法围绕学习多个时间尺度的动作和控制，这种方法将从一开始就考虑这类数据的时变性质。

This project falls within the EPSRC Robotics and Artificial Intelligence Technologies research areas (Engineering theme).As I start my PhD journey, there are two broad ideas, which I believe to be linked, which I am working towards contributing towards and developing further. The first is to develop methods to perform hierarchical reinforcement learning. Reinforcement learning (RL) is a machine learning/artificial intelligence methodology which has been researched since the mid-20th century and yet which has only had marked success in the last four or so years, perhaps most notably in 2016 when an RL-based agent beat the world Go champion Lee Sedol in a tournament reported around the world. At the Oxford Robotics Institute, we are very interested in applying RL to physical robotic control, and area where it is yet to see really impactful success. A limitation of RL is that it is immensely slow and data-intensive to train, and results in very brittle control policies that tend to perform badly on anything other than the exact problem they have learnt to solve. These limitations hold back the application of this exciting methodology in the exciting domain of robotics, and my aim is to help tackle this by building a method to perform hierarchical RL. This would introduce a new structure to RL in which policies and learning would occur and multiple timescales, or levels of 'temporal abstraction'. This would draw on ideas from the more established and successful domain of control theory, in which hierarchical control has long been applied, but bring these ideas into our new, learning based paradigm.Secondly, I hope that through work on learning to control at different levels of temporal abstraction, I will be able to devise a robotics-centric approach to encoding optimally computed robot motions in such a way that they can be automatically permuted and reused across different scenarios and different robots. One of my supervisors, Ioannis Havoutis, is involved in a European project called MEMMO (memory of motion), which is seeking to find a way of performing motion generation for complex robots with arms and legs, based on pre-existing optimally computed and saved (in the 'memory of motion') motion sequences. Crucially, the method sought should allow these sequences to have been designed for or executed on different robotic platforms to the one that will then use the MEMMO system for motion. This requires a solution to the problem of encoding in some way the helpful information from the motions such that it can be drawn on by another system. Current approaches do not consider adequately how best to do this in light of the intended application, instead drawing on conventional approaches such as autoencoding or other methods of dimensionality reduction to 'encode' these motions. This gives us no more advantage when it comes to drawing on, altering and reusing these motions in new situations. Rather, these approaches are simply data compression. Contrastingly, this is a problem I hope to tackle through developing a mature methodology around learning motions and control at multiple timescales, an approach which would consider from the very first the time-varying nature of this sort of data.