NRI: Collaborative Research: Sketching Geometry and Physics Informed Inference for Mobile Robot Manipulation in Cluttered Scenes

NRI：协作研究：在杂乱场景中绘制移动机器人操纵的几何和物理知情推理

• 批准号: 1638060
• 负责人: Joseph LaViola
• 金额: $ 28.64万
• 依托单位: The University of Central Florida Board of Trustees
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-09-01 至 2020-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1638060&HistoricalAwards=false
• 关键词: NRI; Collaborative; Research; Sketching; Geometry

The goal of this project is to improve the ability of robots to manipulate and interact with objects, such as when assisting people to support their daily activities. The key idea is that people can provide robots with important information about their environment and the objects within their environment. In particular, people can use their cognitive skills to name objects, provide an understanding of the geometrical structure of objects, and describe an object's behavior in relation to other objects. Specifically, the project will develop a natural user interface that enables people to provide such information by drawing and sketching on top of the robot's view of the world. Physical simulation will then be used to fill in the missing gaps needed for a robot to complete autonomous manipulation tasks. Thus, the project aims to combine object sketching and physical simulation to better support mobile manipulation tasks as well as learn to perform new manipulation tasks when encountered. The project will support a "Put That There" task, where a user can simply give high-level manipulation commands, with the robot filling in the details necessary to complete the task in a cluttered environment.This project aims to improve goal-directed dexterous robotic manipulation in cluttered and unstructured environments through sketching and physical simulation. Robots operating in human environments face considerable uncertainty in perception due to physical contact and occlusions between objects. This project will address such perceptual uncertainty by combining methods for probabilistic inference with natural sketch-based interfaces to extract, label, and automatically infer the geometry, pose, and behavior of objects in complicated scenes. From a human usability perspective, the project addresses how to best create a sketching language and interfaces for intuitive human-in-the-loop extraction of object geometries and behavior from robot sensing. The planned exploration into sketching methods will also explore what underlying representations, raw point clouds, RGB images and video, or RGBD images will be most conducive to supporting accurate geometry extraction and grasp location identification. Given sketched objects, the project will develop probabilistic physically plausible methods for scene estimation that will enable perception for manipulation in cluttered environments. These methods build upon advances in physical simulation to constrain scene estimates to only plausible configurations to both improve estimation accuracy and enable computational tractability. The project will also develop a "Put That There" testbed using a tablet-based web application to support exploration of these concepts as well as act as user studies to evaluate geometry extraction accuracy and the robustness of physics-based scene estimation algorithms.

该项目的目标是提高机器人操纵物体和与物体互动的能力，例如在帮助人们支持他们的日常活动时。关键的想法是，人们可以向机器人提供关于他们的环境和他们环境中的物体的重要信息。特别是，人们可以使用他们的认知技能来命名对象，提供对对象的几何结构的理解，并描述对象相对于其他对象的行为。具体地说，该项目将开发一个自然的用户界面，使人们能够通过在机器人的世界观之上绘制和绘制草图来提供此类信息。然后，将使用物理模拟来填补机器人完成自主操作任务所需的缺失空白。因此，该项目旨在将对象草图和物理模拟相结合，以更好地支持移动操作任务，并学习在遇到新的操作任务时执行新的操作任务。该项目将支持一个“Put That There”任务，用户只需发出高级操作命令，机器人就可以填写在杂乱的环境中完成任务所需的细节。该项目旨在通过草图绘制和物理模拟来提高杂乱和非结构化环境中目标导向的灵活机器人操作。在人类环境中运行的机器人由于物体之间的物理接触和遮挡而面临着相当大的感知不确定性。该项目将通过将概率推理方法与基于草图的自然界面相结合来解决这种感知不确定性，以提取、标记和自动推断复杂场景中对象的几何、姿势和行为。从人类可用性的角度来看，该项目致力于如何最好地创建一种草图语言和界面，用于直观地从机器人感应中提取对象的几何形状和行为。计划中的草图绘制方法探索还将探索哪些底层表示、原始点云、RGB图像和视频或RGBD图像将最有助于支持准确的几何提取和掌握位置识别。在给定草图对象的情况下，该项目将开发概率的物理上可信的场景估计方法，使人们能够在杂乱的环境中感知操纵。这些方法建立在物理模拟的基础上，将场景估计限制为仅合理的配置，从而既提高了估计精度，又实现了计算处理。该项目还将开发一个使用基于平板电脑的网络应用程序的“Put That There”试验台，以支持对这些概念的探索，并作为用户研究，以评估基于物理的场景估计算法的几何提取精度和稳健性。

项目成果

Sketching Affordances for Human-in-the-loop Robotic Manipulation Tasks

• 发表时间: 2019
• 作者: Sina Masnadi;J. Laviola;Jana Pavlasek;Xiaofang Zhu;Karthik Desingh;O. C. Jenkins

Gemsketch: Interactive Image-Guided Geometry Extraction from Point Clouds

• DOI: 10.1109/icra.2018.8460532
• 发表时间: 2018-05
• 期刊: 2018 IEEE International Conference on Robotics and Automation (ICRA)
• 作者: M. Maghoumi;J. Laviola;Karthik Desingh;O. C. Jenkins

Pitch Pipe: An Automatic Low-pass Filter Calibration Technique for Pointing Tasks

音调管：用于指向任务的自动低通滤波器校准技术

• 发表时间: 2019
• 期刊: Proceedings of Graphics Interface 2019
• 作者: Taranta, Eugene M.;Koh, Seng Lee;Williamson, Brian M.;Pfeil, Kevin P.;Pittman, Corey R.;LaViola, Joseph J.
• 通讯作者: LaViola, Joseph J.

AffordIt!: A Tool for Authoring Object Component Behavior in Virtual Reality

• DOI: 10.20380/gi2020.34
• 发表时间: 2020-04
• 作者: Sina Masnadi;Andrés N. Vargas González;Brian M. Williamson;J. Laviola

Jackknife: A Reliable Recognizer with Few Samples and Many Modalities

Jackknife：一种样本少、模态多的可靠识别器

• DOI: 10.1145/3025453.3026002
• 发表时间: 2017
• 期刊: 2017 CHI Conference on Human Factors in Computing Systems
• 作者: Taranta II, Eugene M.;Samiei, Amirreza;Maghoumi, Mehran;Khaloo, Pooya;Pittman, Corey R.;LaViola Jr., Joseph J.
• 通讯作者: LaViola Jr., Joseph J.