PHYDL: Physics-informed Differentiable Learning for Robotic Manipulation of Viscous and Granular Media

粘性和颗粒状介质在我们的日常生活中无处不在，从面团或豆类食物到混凝土、土壤或沙子等建筑材料。人类可以用手轻松地将沙子变成任何形状；厨师可以在烹饪时用适当的工具操纵面团；建筑工人可以控制各种类型的机械来开采、转移或堆放岩石和土壤。除此之外，还有许多其他涉及操纵颗粒媒体的活动，包括灾难救援、太空探索、水下探测、农业等等。因此，改进技术，使之能够以适用的方式自动操纵这类物质，对我们社会的许多地方都是有益的，但目前这是机器人控制中的一大挑战。传统的机器人运动规划关注于安全和最优的轨迹生成，假设环境中的物体是刚体，即物体只能移动或旋转，而不能变形。复杂的可变形几何特征，以及由于粘性和粒度导致的材料变形的高度不可预测性，使得传统的机器人运动规划无法直接应用，因为传统的机器人运动规划通常由于需要显式设计的刚体模型而无法扩展。基于深度人工神经网络和通过与环境交互以实现特定目标的智能代理进行学习的技术，称为深度强化学习(DRL)，在复杂环境中的运动规划和决策中变得更加流行，而不需要明确地建模环境。DRL通过奖励想要的行为和/或惩罚不想要的行为来训练代理或机器人，这样DRL代理将学会解释它对环境的感知，并通过试验和错误采取最佳行动。通常，DRL代理是在真实的仿真环境中训练的，而不是部署真实的机器人直接与真实世界环境交互。但是，大多数模拟器仅支持刚体环境。另一方面，用于模拟此类材料的数值模型通常在计算上是不可行的，并且对于有效的DRL来说是不切实际的。此外，DRL需要机器人或智能体通过大量随机选择的动作来探索环境，以便从通常效率很低且不安全的奖励或惩罚中学习。为了解决上述问题，该项目将首次将一种名为可微物理的新技术引入机器人智能体的学习和控制循环中，从而解锁一个变革性的机器人学习框架。这种基于物理的可微数值模拟将极大地加快模拟过程，同时另一方面允许我们直接计算最优的物理上看似合理的动作，而不需要探索所有可能的无限动作。换句话说，我们将利用可微性来计算物理上看似合理的有界动作，这将减少动作探索的随机性，从而允许机器人更有效地学习。这一最新趋势在机器人轨迹规划和可微物理学等领域引起了越来越多的关注。该项目将开启一个新的机器人学习框架，用于高效、物理可信和安全的深度强化学习，供自主机器人学习操纵粘性和颗粒状材料。

Viscous and granular media are ubiquitous in our daily life, ranging from food like dough or beans to construction materials like concrete, soils, or sand. Humans can use their hands to transform sands to any shape with ease; cooks can manipulate dough while cooking with proper tools; construction workers can control various types of machinery to mine, transfer or pile rocks and soil. Apart from that, there exist many other activities that involve manipulating granular media, including disaster rescue, space exploration, underwater exploration, agriculture and so forth. As a result, improving techniques that can enable automatic manipulation of such substances in an applicable way is rewarding for many parts of our society, but, currently, it is a big challenge in robotic control.Conventional robot motion planning for manipulation focuses on safe and optimal trajectory generation with the assumption of rigid bodies of objects in the environment, that is, objects can only move or rotate but not deform. The intricate deformable geometric features and consequently the high unpredictability of material deformation due to the viscosity and granularity would prohibit direct applications of traditional robotic motion planning that is usually not scalable for such problems due to the requirement of explicitly designed models for rigid bodies. Techniques based on deep artificial neural networks and learning through intelligent agents interacting with the environment to achieve specific goals, known as Deep Reinforcement Learning (DRL), have become more popular for motion planning and decision making in complex environments without explicitly modelling the environment. DRL trains an agent or a robot by rewarding desired behaviours and/or punishing undesired ones, such that the DRL agent will learn to interpret its environment perception and take optimal actions through trial and error. Usually, the DRL agent is trained in a realistic simulation environment without deploying a real robot to interact with the real-world environment directly. However, most simulators only support rigid-body environments. On the other hand, numerical modelling for simulating such materials is usually computationally prohibitive and impractical for efficient DRL. Moreover, DRL requires a robot or agent to explore the environment with a large number of randomly selected actions in order to learn from getting rewards or penalties that are usually highly inefficient and unsafe.To address the above issues, this project will, for the first time, unlock a transformative robot learning framework by introducing a new technique, named differentiable physics, into the learning and control loop of the robot agent. This differentiable physics-based numerical simulation would greatly accelerate the simulation process, while on the other hand allow us to directly compute optimal physically-plausible actions without exploring all possible actions that are infinitely unbounded. In other words, we will leverage the differentiability nature for calculating physically-plausible bounded actions, which will reduce the amount of randomness for action exploration and hence allow a robot to learn more efficiently. This recent tendency has attracted increasing attention in different communities such as robot trajectory planning and differentiable physics. This project will unlock a new robot learning framework for highly efficient, physically-plausible, and safe deep reinforcement learning for autonomous robots to learn to manipulate viscous and granular materials.