CCRI: Planning: A Community-Standard, Large-Scale Synthetic 3D Scene Dataset for Scene Analysis and Synthesis

CCRI：规划：用于场景分析和合成的社区标准、大规模合成 3D 场景数据集

• 批准号: 2016532
• 负责人: Daniel Ritchie
• 金额: $ 5万
• 依托单位: Brown University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2024-03-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2016532&HistoricalAwards=false
• 关键词: CCRI; Planning; Community; Standard; Large

To function as useful household assistants, robots need to understand what they are seeing and how to navigate in indoor environments. The current state-of-the-art approaches for solving these problems rely on machine learning, and in particular deep learning, which requires large quantities of labeled data (e.g. many images with per-pixel labels indicating what type of object is present at that pixel). Rather than asking people to laboriously label data captured from real-world spaces, a promising alternative approach is to use *synthetic* 3D scenes: virtual 3D models of indoor spaces. The 3D objects which populate these virtual spaces can be equipped with information such as their object type, which allows large sets of labeled training data to be created essentially “for free.” This project aims to construct *the* community-standard, large-scale synthetic 3D scene dataset. While some synthetic 3D scene datasets exist, they are either too small, or they have been subject to onerous use restrictions (and even lawsuits) due to copyright issues on their 3D models, which typically come from for-profit companies. This project will construct a large-scale dataset out of freely-available 3D content. The main contribution of the project is not just this dataset, but also a *scalable pipeline* for creating such 3D scene datasets. This pipeline will be released as open source, allowing others to expand the dataset or to construct their own datasets for needs which may be difficult to anticipate today. In total, the results of this project will enable any researcher (not just those at heavily-resourced institutions) to build AI systems which leverage large-scale synthetic indoor training data.The planned dataset construction pipeline will construct 3D scenes based on 2D floor plan datasets, which already exist at large scale. Using a machine-learning-based system previously developed by the investigators, these 2D floor plans will be converted to 3D models of empty houses. Then, each room in the house will be populated with objects in a plausible arrangement. Initially, this step will be performed by crowd workers on a platform such as Amazon Mechanical Turk. The workers will be instructed to place objects so as to match a photograph, where the photograph is chosen such that its (estimated) room geometry matches the geometry of the empty room to be populated. In a later stage of the project, rooms populated in this manner will be used to train a machine learning model which can automatically place objects based on an input photograph, thus further accelerating the dataset construction process.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

为了充当有用的家务助理，机器人需要了解它们看到的是什么，以及如何在室内环境中导航。当前用于解决这些问题的最先进的方法依赖于机器学习，尤其是深度学习，这需要大量的标记数据(例如，具有指示在该像素处存在什么类型的对象的每像素标签的许多图像)。与其让人们费力地标记从现实世界中捕捉到的数据，另一种有前景的方法是使用合成的3D场景：室内空间的虚拟3D模型。填充这些虚拟空间的3D对象可以配备诸如它们的对象类型之类的信息，这使得基本上可以“免费”地创建大量已标记的训练数据。该项目旨在构建符合社区标准的大规模合成3D场景数据集。虽然存在一些合成3D场景数据集，但它们要么太小，要么由于3D模型的版权问题而受到繁重的使用限制(甚至诉讼)，这些3D模型通常来自营利性公司。该项目将利用免费提供的3D内容构建一个大规模的数据集。该项目的主要贡献不仅是这个数据集，而且还是一个用于创建这样的3D场景数据集的“可伸缩管道”。这条管道将以开源形式发布，允许其他人扩展数据集或构建自己的数据集，以满足今天可能难以预测的需求。总而言之，该项目的结果将使任何研究人员(不仅仅是那些资源丰富的机构的研究人员)都能够建立利用大规模合成室内训练数据的人工智能系统。计划中的数据集建设管道将基于已经大规模存在的2D平面图数据集构建3D场景。使用调查人员之前开发的基于机器学习的系统，这些2D平面图将被转换为空房子的3D模型。然后，房子里的每个房间都将以看似合理的排列方式填充物品。最初，这一步将由Amazon Machine Turk等平台上的众包工作人员执行。工人将被指示放置物体以匹配照片，其中照片的选择使其(估计的)房间几何形状与要填充的空房间的几何形状相匹配。在项目的后期阶段，以这种方式填充的房间将用于训练机器学习模型，该模型可以根据输入照片自动放置对象，从而进一步加快数据集构建过程。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。