NRI: FND: Using Multi-Modal Data to Make Robotic Grasp Algorithms Aware of Human Preferences for Safe Collaborative Robot-Human Handover Interactions with Novel Objects

该项目通过使带有操纵臂的机器人了解人类与物体交互的偏好，为促进安全的机器人与人协作切换做出了贡献。在诸如医疗设施、仓储、零售、发动机维修和飞机组装等环境中，机器人可能被期望与人类合作以成功完成任务，机器人操纵器必须移交物体，使得人们可以最佳地保持它们，而不必担心物体掉落或人被夹持器或手臂伤害，并且没有对象不可达的不便。为了实现安全抓取，该项目将提供算法，该算法使用从多个视角捕获的人类与物体交互的数据，以自动预测人类抓取物体的首选位置，物体与人的最佳方向和距离，以及机械手抓取器释放物体的安全点。该研究团队将深入到北部地区技术机会有限的两年制和四年制大学，为来自代表性不足社区的女性和学生提供研究机会。该项目通过对人们相互之间以及与环境中的物体互动时的自然行为进行多模态传感，提供全面的细粒度洞察，从而推进无处不在的合作机器人的研究。该项目实现了三个目标，以解决传播人类交接偏好的理解到大量的新的野外对象的集合的协作机器人的可定制性，以新的环境中的差距差距。首先，研究团队将收集一个大型的多视角多模态数据集，并对收集到的数据进行实证分析，以了解持有位置，结束姿势和释放点的偏好，使用对象呈现的主题评级。使用的模式将包括深度相机，以获得对物体几何形状和空间关系的理解，以及热成像相机，以根据传递到物体表面的热量分析人体接触的位置。这项工作将提供一个定量的人类偏好的几何形状和功能的对象方面的移交参数的分解。其次，该团队将创建基于概率模型的感知算法，以使用对象的深度图像作为输入来预测按偏好排序的切换参数。这项工作使合作机器人能够像人类一样意识到偏好的多样性，以及人们分配给交互的优先级。第三，该团队将提供机器人操纵器，这些机器人操纵器使用经过训练的感知算法对新物体进行交接操作，同时意识到人类的行为。该项目活动的成功完成将使机器人机械手能够快速传播人类对新物体和环境的交接行为，以提高合作机器人的社会接受度。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project contributes advancements in promoting safe collaborative robot-to-human handovers by making robots with manipulator arms aware of human preferences for interactions with objects. In environments such as healthcare facilities, warehousing, retail, engine repair, and aircraft assembly, where robots may be expected to collaborate with humans for successful accomplishment of tasks, it is essential that robotic manipulators hand over objects such that people can optimally hold them, without fear of the object falling or the person being injured by the gripper or arm, and without the inconvenience of the object being unreachable. To enable safe handovers, the project will provide algorithms that use data on human interactions with objects captured from multiple viewpoints to automatically predict preferred locations of human grasp on objects, optimal orientation and distance of the object from the person, and safe point of release of the object by manipulator grippers. The research team will reach out to two-year and four-year colleges with limited technological opportunities in the North Country to provide research opportunities to women and students from underrepresented communities.The project advances research in ubiquitous co-robots by providing holistic fine-grained insight through multi-modal sensing on natural behaviors of people as they interact with each other and with objects in their environments. The project accomplishes three objectives to address the gap on propagating understanding of human handover preferences to large collections of novel in-the-wild objects for customizability of co-robots to new environments. First, the research team will collect a large multi-viewpoint multi-modal dataset on two-person handovers and perform empirical analysis of the collected data to understand preferences on hold locations, end pose, and release point using subject ratings of object presentations. Modalities used will consist of depth cameras to acquire understanding on object geometry and spatial relationships, and thermal cameras to analyze locations of human contact based on heat transferred to object surfaces. This work will provide a quantitative decomposition of human preferences for handover parameters in terms of geometric form and functionality of objects. Second, the team will create perception algorithms based on probabilistic models to perform prediction of handover parameters ranked in order of preference using depth images of objects as input. This work enables equipping co-robots with human-like awareness of diversity in preferences, and the priorities that people assign to interactions. Third, the team will provide robotic manipulators that use the trained perception algorithms to perform handover manipulations on novel objects while being aware of human behavior. Successful accomplishment of the project activities will enable rapid propagation of robotic manipulators aware of human handover behavior to new objects and environments for enhanced social acceptability of co-robots.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.