NRI: FND: Using Multi-Modal Data to Make Robotic Grasp Algorithms Aware of Human Preferences for Safe Collaborative Robot-Human Handover Interactions with Novel Objects

该项目通过使具有机械手手臂的机器人意识到人类对与物体交互的偏好来促进安全协作的机器人到人类的移交方面的进步。在医疗设施、仓储、零售、发动机维修和飞机组装等环境中，机器人可能会与人类合作成功完成任务，因此机器人机械手交接对象以使人能够最佳地握住它们是至关重要的，而不必担心对象坠落或被夹持器或手臂伤害，也不必担心对象无法触及的不便。为了实现安全移交，该项目将提供算法，使用从多个视点捕获的人类与物体相互作用的数据来自动预测人类抓取物体的首选位置、物体与人的最佳方向和距离以及机械手抓手释放物体的安全点。该研究团队将联系北部国家技术机会有限的两年制和四年制大学，为来自代表性不足社区的女性和学生提供研究机会。该项目通过对人类相互作用以及与环境中的物体互动时的自然行为进行多模式传感，提供全面的细粒度洞察，从而推进无处不在的协作机器人的研究。该项目实现了三个目标，以解决在将对人类移交偏好的理解传播到大量新的野外对象集合方面的差距，以实现协作机器人对新环境的定制。首先，研究团队将收集关于两人移交的大型多视点多模式数据集，并对收集的数据进行实证分析，以了解使用对象呈现的主题评级对保留位置、结束姿势和放行点的偏好。使用的模式将包括深度相机和热相机，前者用于了解物体的几何形状和空间关系，后者用于根据转移到物体表面的热量来分析人类接触的位置。这项工作将根据物体的几何形状和功能对人类对切换参数的偏好进行量化分解。其次，该团队将创建基于概率模型的感知算法，以对象的深度图像作为输入，执行对按偏好排序的切换参数的预测。这项工作使协作机器人能够像人类一样意识到偏好的多样性，以及人们赋予交互的优先级。第三，该团队将提供机器人机械手，使用训练好的感知算法在新对象上执行移交操作，同时意识到人类的行为。项目活动的成功完成将使机器人操作员能够将意识到人类移交行为的机器人快速传播到新的对象和环境，以增强合作机器人的社会可接受性。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project contributes advancements in promoting safe collaborative robot-to-human handovers by making robots with manipulator arms aware of human preferences for interactions with objects. In environments such as healthcare facilities, warehousing, retail, engine repair, and aircraft assembly, where robots may be expected to collaborate with humans for successful accomplishment of tasks, it is essential that robotic manipulators hand over objects such that people can optimally hold them, without fear of the object falling or the person being injured by the gripper or arm, and without the inconvenience of the object being unreachable. To enable safe handovers, the project will provide algorithms that use data on human interactions with objects captured from multiple viewpoints to automatically predict preferred locations of human grasp on objects, optimal orientation and distance of the object from the person, and safe point of release of the object by manipulator grippers. The research team will reach out to two-year and four-year colleges with limited technological opportunities in the North Country to provide research opportunities to women and students from underrepresented communities.The project advances research in ubiquitous co-robots by providing holistic fine-grained insight through multi-modal sensing on natural behaviors of people as they interact with each other and with objects in their environments. The project accomplishes three objectives to address the gap on propagating understanding of human handover preferences to large collections of novel in-the-wild objects for customizability of co-robots to new environments. First, the research team will collect a large multi-viewpoint multi-modal dataset on two-person handovers and perform empirical analysis of the collected data to understand preferences on hold locations, end pose, and release point using subject ratings of object presentations. Modalities used will consist of depth cameras to acquire understanding on object geometry and spatial relationships, and thermal cameras to analyze locations of human contact based on heat transferred to object surfaces. This work will provide a quantitative decomposition of human preferences for handover parameters in terms of geometric form and functionality of objects. Second, the team will create perception algorithms based on probabilistic models to perform prediction of handover parameters ranked in order of preference using depth images of objects as input. This work enables equipping co-robots with human-like awareness of diversity in preferences, and the priorities that people assign to interactions. Third, the team will provide robotic manipulators that use the trained perception algorithms to perform handover manipulations on novel objects while being aware of human behavior. Successful accomplishment of the project activities will enable rapid propagation of robotic manipulators aware of human handover behavior to new objects and environments for enhanced social acceptability of co-robots.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.