Collaborative Research: Visual Tactile Neural Fields for Active Digital Twin Generation

当机器人能够快速地将自己的感官数据与环境模型结合起来，就像人类本能地利用自己所有的感官和知识来完成日常任务一样，它们在日常活动中会表现得更好。然而，机器人必须通过编程来创建这些模型，而这些模型是人类凭直觉、毫不费力和稳健地完成的。这个机器人项目探索了一种新的算法方法，将视觉和触觉感官数据与物理知识和学习能力相结合，使机器人的规划和推理更加有效、高效和适应性强。该项目包括研究原型的开发和测试，新课程的准备，以及向高中学生和教师以及公众推广。这个项目引入了一种新的数据表示，称为视觉触觉神经场（VTNF），它允许机器人将来自视觉和触觉传感器的数据结合起来，创建一个物体的统一模型。VTNF被设计成以闭环方式使用，其中机器人可以使用来自其与对象的物理交互的数据来创建或改进模型，并且可以使用其当前对模型的理解来告知如何最好地与物理对象交互。为此，研究人员创建了生成VTNF模型所需的数学技术、计算工具和机器人硬件。研究人员还开发了量化物体不确定性的技术，并利用这种不确定性来学习搜索策略，从而使机器人能够尽快生成准确的模型。VTNF允许轻松添加关于对象的新属性，为其他研究人员和实践者提供了灵活的表征基础，使机器人能够通过更详细地了解周围环境及其与之的相互作用来更快地学习。该项目由机器人跨部门基础研究计划和国家机器人计划支持，由工程（ENG）和计算机与信息科学与工程（CISE）共同管理和资助。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Robots will perform better at everyday activities when they can quickly combine their sensory data into a model of their environment, just like how humans instinctively use all their senses and knowledge to accomplish daily tasks. Robots, however, must be programmed to create these models that humans do intuitively, effortlessly, and robustly. This robotics project explores a novel algorithmic approach that combines visual and tactile sensory data with a knowledge of physics and a capability to learn that makes robot planning and reasoning more effective, efficient, and adaptable. The project includes the development and testing of research prototypes, preparation of new curriculum, and outreach to high school students and teachers and to the general public.This project introduces a new data representation, called a Visual Tactile Neural Field (VTNF), that allows robots to combine data from visual and tactile sensors to create a unified model of an object. The VTNF is designed to be used in a closed-loop manner, where a robot may use data from its physical interactions with an object to create or improve a model and may use its current understanding of a model to inform how best to interact with a physical object. Towards this end, the investigators create the mathematical techniques, computational tools, and robot hardware necessary to generate a VTNF model. The investigators also develop techniques to quantify the uncertainty about an object and use this uncertainty to learn search policies that allow robots to generate accurate models as quickly as possible. The VTNF, which allows for the easy addition of new properties about an object, provides a flexible representational foundation for other researchers and practitioners to use to enable robots to learn faster by having a more detailed understanding of both the surrounding environment and their interactions with it.This project is supported by the cross-directorate Foundational Research program in Robotics and the National Robotics Initiative, jointly managed and funded by the Directorates for Engineering (ENG) and Computer and Information Science and Engineering (CISE).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.