CAREER: Visual Manipulation Learning for Challenging Object Grasping

该学院早期职业发展(CALEAR)项目旨在显著提高机器人系统抓取物体的能力。物体抓取是完成各种操作任务的重要前提。人类能够灵活地抓住不同的物体，即使他们的工作空间杂乱无章。当一个物体在一个受限的空间中时，例如纸箱或架子，人类通常会通过将物体推向墙壁或角落来抓住它，从而利用这种限制。即使目标对象隐藏在一堆对象中，人类也会通过移除杂乱的对象来主动寻找它。虽然在这种具有挑战性的场景中抓取物体对人类来说似乎是毫不费力和自然的，但目前的机器人仍然在适度杂乱的桌面环境中抓取物体。这个项目将开发计算算法，使机器人能够执行这种具有挑战性的对象抓取。这个项目有可能拓宽机器人操作的应用领域，例如柔性制造(例如，通过提供正确的部件来支持人类工人)、农业(例如，自动化水果和蔬菜收获)、仓库履行中心或杂货购物(例如，在货架或垃圾箱中挑选和放置订购的物品)以及老年护理(例如，取出遥控器或药丸)，这些都是当前机器人实验室无法实现的。该项目的目标是开发新的计算算法，使机器人能够视觉理解场景，学习执行正确的操作动作序列，并适应不同的环境设置。这个项目将解决机器人物体抓取中的三个基本挑战：(1)上下文感知的物体抓取--考虑与抓取相关的空间上下文，如杂乱、可达性和碰撞；(2)通过利用夹具抓取物体--利用环境夹具通过学习像素级或物体级的承受能力来抓取具有挑战性的物体；以及(3)物体搜索和抓取--搜索和抓取通过参考图像或人类自然语言查询的隐藏目标物体。所有的算法和系统都被设计成自我监督的，这意味着它们可以学习和适应新的对象、环境和任务，只需最少的人工干预或指导。该项目将为我们社会的广泛领域，如家庭、工厂、农场、仓库和老年护理设施，提供负担得起和可靠的机器人劳动力奠定坚实的基础。该项目由跨部门机器人基础研究计划支持，该计划由工程总监(ENG)和计算机和信息科学与工程(CEISE)共同管理和资助。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This Faculty Early Career Development (CAREER) project seeks to significantly enhance the capabilities of robotic systems to grasp objects. Object grasping is an important prerequisite for various manipulation tasks. Humans are capable of grasping diverse objects dexterously even if their workspaces are cluttered. When an object is in a constrained space, such as a cardboard box or a shelf, humans often take advantage of the constraint by pushing the object toward a wall or a corner to grasp it. Even if a targeted object is hidden in a pile of objects, humans actively search for it by removing the clutter objects. While object grasping in such challenging scenarios seems effortless and natural for humans, current robots are still grasping objects in moderately cluttered tabletop environments. This project will develop computational algorithms to allow robots to perform such challenging object grasping. This project has the potential to broaden the application domains of robotic manipulation, such as flexible manufacturing (e.g., supporting human worker by providing right parts), agriculture (e.g., automated fruit and vegetable harvesting), warehouse fulfillment centers or grocery shopping (e.g., pick-and-place ordered items in shelves or bins), and eldercare (e.g., fetching a remote controller or pills), which have not been feasible with current robotic labor.The objective of this project is to develop novel computational algorithms that enable robots to visually understand scenes, learn to perform a proper manipulation action sequence, and adapt to different environmental settings. This project will address three fundamental challenges in robotic object grasping: (1) context-aware object grasping -- considering spatial contexts related to grasping, such as clutteredness, reachability, and collision, (2) object grasping via leveraging fixtures -- making use of environmental fixtures to grasp challenging objects by learning pixel-level or object-level affordances, and (3) object searching and grasping -- searching and grasping a hidden target object queried by either a reference image or a human natural language. All the algorithms and systems are designed to be self-supervised, meaning that they can learn and adapt to novel objects, environments, and tasks with minimal human interventions or guidance. This project will lay a solid foundation for the affordable and reliable robotic labor beneficial to the broad areas of our society, such as homes, factories, farms, warehouses, and eldercare facilities.This project is supported by the cross-directorate Foundational Research in Robotics program, jointly managed and funded by the Directorates for Engineering (ENG) and Computer and Information Science and Engineering (CISE).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.