CAREER: Learning and Sharing Transferable Grounded Object Knowledge for Collaborative Robots

视觉和非视觉传感技术(例如人工触觉)的进步使机器人能够极大地提高其对象操纵技能。了解物体的移动、声音和手感可以改善人与机器人在制造环境中组装组件或在仓库和配送中心对物体进行分拣等任务中的协作。然而，一个机器人学习的对象知识不容易被具有不同身体、传感器和运动动作的不同机器人使用。在实践中，这意味着当部署一个新的机器人时，它必须从头开始学习许多技能和许多知识。这个学院早期职业发展(CALEAR)项目将开发在机器人之间传递对象知识的方法，以便新部署的机器人能够理解在相同或相似环境中运行的其他机器人的经验。该项目将促进家庭和工作场所中的协作机器人感知和推理对象的属性的能力。在辅助环境中的机器人将更好地学习需要触觉的任务，例如，帮助残疾人脱鞋。该项目还将提高机器人将语言与视觉和非视觉感知联系起来的能力，例如，帮助机器人识别特定对象可以被称为“软”，这在人类和机器人使用语言交流对象时非常重要。多感知对象知识包括以多种感官模式为语言奠定基础的识别模型(例如，根据按下对象时产生的触觉读数识别对象是否“软”的分类器)，以及预测机器人环境因其动作而发生变化的向前模型。该项目的研究目标是使多个异质机器人能够学习和共享多感官对象知识，以减少每个机器人需要收集的交互数据量。该项目假设，具有不同实施例和传感器的两个或多个机器人可以通过使用共享嵌入空间来学习传递多感觉表征，机器人将自己的经验映射到该空间，并使用其他机器人的经验从中学习。这项研究将开发这种转移的理论框架，以及可扩展到大量机器人、感觉模式、对象和交互行为的算法和表示法。实验评估将在一开始使用现有的数据集，以及将通过多个机器人平台收集的日益复杂的新数据集。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Advances in visual and non-visual sensing technologies (e.g., artificial sense of touch) have enabled robots to greatly improve their object manipulation skills. Understanding how objects move, sound, and feel like can improve human-robot collaboration in tasks such as assembling components in manufacturing environments or sorting objects in warehouses and distribution centers. However, learned object knowledge by one robot cannot easily be used by a different robot, with a different body, sensors, and movement actions. In practice, this means that when a new robot is deployed, it has to learn many of its skills and much of its knowledge from scratch. This Faculty Early Career Development (CAREER) project will develop methods for transferring object knowledge across robots so that a newly deployed robot can make sense of the experiences of other robots that have operated in the same or similar environments. This project will facilitate the ability of collaborative robots in homes and workplaces to perceive and reason about the properties of objects. Robots in assistive settings will be better at learning tasks that require the sense of touch, for example, helping a disabled person take off their shoes. The project will also improve robots’ ability to connect language to visual and non-visual perception, for example, helping robots recognize that a particular object can be referred to as “soft”, which is important when humans and robots use language to communicate about objects.Multisensory object knowledge includes recognition models that ground language in multiple sensory modalities (e.g., a classifier that recognizes if an object is “soft” given haptic readings produced when pressing the object) as well as forward models which predict changes in the robot’s environment as a result of its actions. The research objective of this project is to enable multiple heterogeneous robots to learn and share multisensory object knowledge to reduce the amount of interaction data each individual robot needs to collect. This project hypothesizes that two or more robots with different embodiments and sensors can learn to transfer multisensory representations through the use of shared embedding spaces, to which robots map their own experiences and from which they learn using the experiences of other robots. This research will develop the theoretical framework for such transfer along with algorithms and representations that scale to large numbers of robots, sensory modalities, objects, and interaction behaviors. Experimental evaluation will be conducted using existing datasets in the beginning, as well as new datasets with increasing complexity that will be collected with multiple robotic platforms.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.