CAREER: Learning and Sharing Transferable Grounded Object Knowledge for Collaborative Robots

视觉和非视觉传感技术的进步（例如，人工触觉）使机器人能够大大提高其物体操纵技能。了解物体的移动、声音和感觉可以改善人机协作，例如在制造环境中组装组件或在仓库和配送中心分拣物体。然而，由一个机器人学习的对象知识不能容易地被具有不同身体、传感器和运动动作的不同机器人使用。在实践中，这意味着当一个新的机器人被部署时，它必须从头开始学习它的许多技能和知识。这个教师早期职业发展（CAREER）项目将开发跨机器人传输对象知识的方法，以便新部署的机器人可以理解在相同或相似环境中操作的其他机器人的经验。该项目将促进家庭和工作场所的协作机器人感知和推理物体属性的能力。辅助环境中的机器人将更善于学习需要触觉的任务，例如帮助残疾人脱鞋。该项目还将提高机器人将语言与视觉和非视觉感知联系起来的能力，例如，帮助机器人识别特定对象可以被称为“软”，这在人类和机器人使用语言就对象进行交流时非常重要。多感官对象知识包括以多种感官模态（例如，识别物体是否“软”的分类器（当按压物体时产生触觉读数）以及预测机器人的环境中的变化作为其动作的结果的前向模型。该项目的研究目标是使多个异构机器人能够学习和共享多感官对象知识，以减少每个机器人需要收集的交互数据量。该项目假设具有不同实施例和传感器的两个或多个机器人可以通过使用共享嵌入空间来学习传递多感官表征，机器人将自己的经验映射到其中，并从其他机器人的经验中学习。这项研究将发展这种转移的理论框架，沿着的算法和表示，可扩展到大量的机器人，感官形式，对象和交互行为。实验评估将使用现有的数据集进行，以及将使用多个机器人平台收集的越来越复杂的新数据集进行。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估而被认为值得支持。

Advances in visual and non-visual sensing technologies (e.g., artificial sense of touch) have enabled robots to greatly improve their object manipulation skills. Understanding how objects move, sound, and feel like can improve human-robot collaboration in tasks such as assembling components in manufacturing environments or sorting objects in warehouses and distribution centers. However, learned object knowledge by one robot cannot easily be used by a different robot, with a different body, sensors, and movement actions. In practice, this means that when a new robot is deployed, it has to learn many of its skills and much of its knowledge from scratch. This Faculty Early Career Development (CAREER) project will develop methods for transferring object knowledge across robots so that a newly deployed robot can make sense of the experiences of other robots that have operated in the same or similar environments. This project will facilitate the ability of collaborative robots in homes and workplaces to perceive and reason about the properties of objects. Robots in assistive settings will be better at learning tasks that require the sense of touch, for example, helping a disabled person take off their shoes. The project will also improve robots’ ability to connect language to visual and non-visual perception, for example, helping robots recognize that a particular object can be referred to as “soft”, which is important when humans and robots use language to communicate about objects.Multisensory object knowledge includes recognition models that ground language in multiple sensory modalities (e.g., a classifier that recognizes if an object is “soft” given haptic readings produced when pressing the object) as well as forward models which predict changes in the robot’s environment as a result of its actions. The research objective of this project is to enable multiple heterogeneous robots to learn and share multisensory object knowledge to reduce the amount of interaction data each individual robot needs to collect. This project hypothesizes that two or more robots with different embodiments and sensors can learn to transfer multisensory representations through the use of shared embedding spaces, to which robots map their own experiences and from which they learn using the experiences of other robots. This research will develop the theoretical framework for such transfer along with algorithms and representations that scale to large numbers of robots, sensory modalities, objects, and interaction behaviors. Experimental evaluation will be conducted using existing datasets in the beginning, as well as new datasets with increasing complexity that will be collected with multiple robotic platforms.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.