AI and robotics - applying the maximum entropy framework to real-world robotics tasks

背景和影响近年来，机器人技术在走出实验室和进入现实世界任务方面显示出越来越大的前景。汽车制造等需要简单和重复运动的领域多年来一直感受到机器人技术的影响，但目前的挑战是将这种影响扩展到涉及人类和机器人之间交互的大型动态环境。一旦大规模部署，智能自主系统的经济影响将是巨大的，使一个人的工作可以被多次利用，并创造出数量级的效率提升。这些系统的关键组件之一是控制，这也是我的研究对象。目标和对象与真实世界的机器人相关的控制任务大致分为两类：运动、我的注意力和操纵。虽然在平坦的地面上移动相当简单，但在崎岖的地形上移动变得困难得多，需要使用视觉等额外的感知模式来预测障碍并采取相应的行动。我研究的主要目的之一是增加最先进的运动控制器的有效运行范围。更复杂的运动形式，如攀登和跳跃，表现得很好，目前还不是现代机器人系统所能达到的。扩展这些能力将极大地增强这些系统的自主性，进一步推动现实世界的部署。研究方法的新颖性支持最先进的机器人学的关键技术之一是深度强化学习(RL)。这将是我的研究中主要关注的技术，并将通过解决以下问题来扩展其能力。第一个问题是如何使训练强化学习系统更稳定和有效。RL算法通常会陷入局部最优解，要么部分解决问题，要么以一种无用的方式对奖励函数进行游戏--就像一个不动的代理，以避免因崩溃但回避我们希望它完成的任务而受到惩罚。此外，搜索有用的操作通常涉及高度不稳定的行为，当这些行为部署在现实世界的系统上时，很容易导致硬件损坏，或者更重要的是，对附近的人造成损害。如果我们希望有一天拥有能够适应环境实时变化的智能自主系统，这些问题必须得到解决。从我的研究来看，最大熵框架(MaxEnt)是解决这两个问题的一个有前途的解决方案。最大熵算法联合优化代理的奖励函数和它所采取的行动分布的“熵”--可以被认为是随机性的衡量标准。这种方法的好处如下。首先，它是一种简单而稳健的勘探-开采权衡解决方案。这是设计RL系统的关键难题之一，可以被认为是平衡我们知道的目前有效的解决方案，并探索可能更有效的解决方案的其他选择。其次，MaxEnt与其他方法相比具有独特的优势，因为它允许多模式解决方案，这意味着如果有多种同样有效的解决问题的方法，代理可以保留所有方法，给我们更大的灵活性。与EPSRC的战略和研究领域保持一致我相信这个项目正好属于UKRI的人工智能和机器人主题，因为健壮的控制策略是我们的系统需要解决的核心挑战之一，从而使英国具有弹性和有效性。此外，我的研究与许多战略优先事项不谋而合，比如人工智能、工程和技术的前沿，甚至可能是更遥远的领域，比如在外科手术中增加机器人操纵器来改变健康和医疗保健。

Context and impactIn recent years, robotics has shown increasing promise in moving outside of laboratories and into real-world tasks. Areas such as car manufacturing that require simple and repetitive motions have felt the impact of robotics for years, but the current challenge is to extend this reach into large, dynamic environments that involve interaction between humans and robots. The economic impact of intelligent autonomous systems, once deployed at scale, will be vast, allowing the work of one person to be leveraged many times over, and creating orders of magnitude of efficiency gains. One of the key components of these systems is control, which is where my research sits.Aims and objectivesThe control tasks relevant to real-world robotics broadly fall into two categories: locomotion, my focus, and manipulation. While locomotion over flat ground is fairly straightforward, things become much more difficult over rough terrain, requiring the use of extra sensory modes like vision to anticipate obstacles and act accordingly. One of the principal aims of my research is to increase the effective range of operation of state-of-the-art locomotion controllers. More complex forms of movement like climbing and jumping, performed robustly, are currently out of the reach of modern robotics systems. Extending these capabilities would greatly enhance the domain of autonomy of these systems, furthering real-world deployment. Novelty of the research methodologyOne of the key technologies powering state-of-the-art robotics is Deep Reinforcement learning (RL). This will be the technique I will primarily focus on for my research and will aim to extend its capabilities by addressing the following questions.The first of these is how we can make training reinforcement learning systems more stable and effective. RL algorithms commonly fall into locally optimal solutions that either partially solve a problem or 'games' the reward function in an unhelpful way - like an agent that doesn't move, to avoid penalties for collapsing but sidestepping the task we want it to accomplish. Additionally, the search for useful actions often involves highly unstable behaviour which when deployed on real-world systems can easily result in damage to the hardware, or more importantly, to nearby people. If we wish to one day have intelligent autonomous systems that can adapt to real-time changes in their environment, these problems must be solved. From my research, a promising solution to both of these issues is the maximum entropy framework (MaxEnt).Maximum entropy algorithms jointly optimise for the agent's reward function and the 'entropy' - which can be thought of as a measure of randomness - of the distribution of actions it takes. The benefits of this approach are as follows. Firstly it is a simple and robust solution to the exploration-exploitation trade-off. This is one of the key dilemmas in designing an RL system and can be thought of as balancing solutions we know are currently effective, and exploring other options for solutions that might be even more effective. Secondly, MaxEnt has a unique advantage over other methods in that it allows for multimodal solutions, meaning that if there are multiple equally valid ways of solving a problem, the agent can retain all of them, giving us greater flexibility.Alignment to EPSRC's strategies and research areasI believe this project falls squarely under the UKRI's AI and robotics theme, as robust control policies are one of the core challenges that need to be solved for our systems, and thus the UK, to be resilient and effective. Additionally, my research coincides with many of the strategic priorities like artificial intelligence, frontiers in engineering and technology, and possibly even more distant areas like transforming health and healthcare with the addition of robotic manipulators in surgical procedures.