RI: Medium: Collaborative Research: Towards Practical Encoderless Robotics Through Vision-Based Training and Adaptation

随着机器人扩展到非结构化和动态的人类环境（如家庭、办公室和医院），它们需要一种新的设计方法。这些机器人需要在人类旁边安全操作；人们期望它们能够处理人类环境中固有的频繁变化和不确定性；它们应该尽可能便宜，以便广泛传播。这些标准导致了柔性机器人、3D打印机器人和廉价的消费级硬件的出现，所有这些都构成了从工业中使用的具有严格公差的重型和刚性机器人的重大转变。传统的测量设备，适合传感和控制刚性机器人的运动，即关节编码器，是不兼容或不切实际的许多这些新型机器人。不依赖于编码器的替代方法在机器人技术中很大程度上是缺失的，必须为这些新颖的设计模型开发。该项目研究了仅使用相机来感知和控制机器人运动的方法。基于视觉的机器人行走、物体抓取和操作算法将被导出。这种算法不仅使新浪潮机器人能够在非结构化环境中使用，而且还将大大降低传统机器人系统的成本，从而促进它们在工业和教育目的上的推广。该项目将重点利用基于视觉的估计方案和学习方法，在一个框架内获取机器人配置信息和任务模型，其中建模不准确性和环境不确定性由鲁棒视觉伺服方法处理。视觉观察将用于建模执行器输入、机械手配置和任务状态之间的关系，并且它们将与基于自适应视觉的控制方案相结合，该方案对建模不确定性和干扰具有鲁棒性。该框架将从根本上依赖于使用卷积神经网络（cnn），仅从观察中构建模型，用于构型的低维表示和机械手的图像分割。强化学习方法也将被应用于评估这种方法与离线学习表征的模块化组合的实用性，以执行复杂的定位和控制任务。这些方法将在手内操作、手术工具控制、3d打印多腿机器人的运动以及使用软连续体机械手的力控制抓取和钉插入的背景下进行评估。我们提出的工作的贡献是不需要机器人配置的先验模型，因为它是预先明确观察和推断的（系统识别）；通过动态调整机器人动力学（从模型到确认）来解决影响任务性能的不确定性；我们的方法的广泛适用性将通过应用于各种各样的平台来证明。该项目的工作将有助于在一系列领域实现更低成本的机器人和机电一体化硬件，特别是将影响控制柔性和欠驱动结构的能力。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

As robots branch out into unstructured and dynamic human environments (such as homes, offices, and hospitals), they require a new design methodology. These robots need to be safe to operate next to humans; they are expected to handle frequent changes and uncertainties that are inherent in human environments; and they should be as inexpensive as possible to enable wide-spread dissemination. Such criteria have lead to the emergence of compliant/soft robots, 3D printed robots, and inexpensive consumer-grade hardware, all of which constitute a major shift from heavy and rigid robots with tight tolerances used in industry. Traditional measurement devices that are suitable for sensing and controlling the motion of the rigid robots, i.e. joint encoders, are incompatible or impractical for many of these new types of robot. Alternative approaches that do not rely on encoders are largely missing from robotics technology and must be developed for these novel design models. This project investigates ways of using only cameras for sensing and controlling the robot's motion. Vision-based algorithms for robotic walking, object grasping and manipulation will be derived. Such algorithms will not only enable the use of the new-wave robots in unstructured environments but will also significantly lower the cost of traditional robotic systems, and therefore, boost their dissemination for industry and educational purposes.The project will focus on utilizing vision-based estimation schemes and learning methods for acquiring both robot configuration information and task models within a framework where modeling inaccuracies and environment uncertainties are dealt with by robust visual servoing approaches. Visual observations will be used to model the relationship between actuator inputs, manipulator configuration, and task states, and they will be combined with adaptive vision-based control schemes that are robust to modeling uncertainties and disturbances. The framework will fundamentally rely on using convolutional neural networks (CNNs) to build the models from observation alone, both for a low-dimensional representation of configuration and for an image segmentation of the manipulator. Reinforcement learning methods will also be applied to assess the practicality of a modular combination of such methods with the offline learned representations to perform complex positioning and control tasks. These approaches will be evaluated in the context of within-hand manipulation, compliant surgical tool control, locomotion of a 3D-printed multi-legged robot, and force-controlled grasping and peg-insertion using a soft continuum manipulator. The contributions of our proposed work are that no prior model of a robot's configuration is needed because it is explicitly observed and inferred up-front (system identification); uncertainty affecting task performance is addressed by adapting the robot dynamics on-the-fly (model-through-confirmation); and the broad applicability of our methods will be demonstrated through application to a wide variety of platforms. Work done on this project will help to enable lower cost robotic and mechatronic hardware across a range of domains and will particularly impact the ability to control compliant and under-actuated structures.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.