CCRI: Planning: A Community-Standard, Large-Scale Synthetic 3D Scene Dataset for Scene Analysis and Synthesis

为了成为有用的家庭助手，机器人需要了解它们所看到的东西，以及如何在室内环境中导航。目前解决这些问题的最先进的方法依赖于机器学习，特别是深度学习，这需要大量的标记数据（例如，许多带有每像素标签的图像，表明该像素上存在什么类型的对象）。而不是让人们费力地标记从现实世界空间捕获的数据，一个有前途的替代方法是使用“合成”3D场景：室内空间的虚拟3D模型。填充这些虚拟空间的3D对象可以配备诸如其对象类型之类的信息，这使得基本上“免费”创建大量标记训练数据。本项目旨在构建* *社区标准的大规模合成3D场景数据集。虽然存在一些合成3D场景数据集，但它们要么太小，要么由于3D模型的版权问题而受到繁重的使用限制（甚至诉讼），这些模型通常来自营利性公司。这个项目将构建一个大规模的数据集，从免费的3D内容。这个项目的主要贡献不仅仅是这个数据集，还有一个可扩展的管道来创建这样的3D场景数据集。这个管道将作为开源发布，允许其他人扩展数据集或构建他们自己的数据集，以满足今天可能难以预测的需求。总的来说，这个项目的结果将使任何研究人员（不仅仅是那些资源丰富的机构的研究人员）能够构建利用大规模室内综合训练数据的人工智能系统。规划的数据集构建管道将基于已经大规模存在的二维平面图数据集构建三维场景。利用先前由研究人员开发的基于机器学习的系统，这些2D平面图将被转换为空房子的3D模型。然后，房子里的每个房间都将以合理的排列方式填充物体。最初，这一步骤将由亚马逊土耳其机械（Amazon Mechanical Turk）等平台上的众工执行。工作人员将被指示放置物体以匹配照片，其中照片的选择使其（估计）房间的几何形状与要填充的空房间的几何形状相匹配。在项目的后期阶段，以这种方式填充的房间将用于训练机器学习模型，该模型可以根据输入的照片自动放置物体，从而进一步加快数据集构建过程。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

To function as useful household assistants, robots need to understand what they are seeing and how to navigate in indoor environments. The current state-of-the-art approaches for solving these problems rely on machine learning, and in particular deep learning, which requires large quantities of labeled data (e.g. many images with per-pixel labels indicating what type of object is present at that pixel). Rather than asking people to laboriously label data captured from real-world spaces, a promising alternative approach is to use *synthetic* 3D scenes: virtual 3D models of indoor spaces. The 3D objects which populate these virtual spaces can be equipped with information such as their object type, which allows large sets of labeled training data to be created essentially “for free.” This project aims to construct *the* community-standard, large-scale synthetic 3D scene dataset. While some synthetic 3D scene datasets exist, they are either too small, or they have been subject to onerous use restrictions (and even lawsuits) due to copyright issues on their 3D models, which typically come from for-profit companies. This project will construct a large-scale dataset out of freely-available 3D content. The main contribution of the project is not just this dataset, but also a *scalable pipeline* for creating such 3D scene datasets. This pipeline will be released as open source, allowing others to expand the dataset or to construct their own datasets for needs which may be difficult to anticipate today. In total, the results of this project will enable any researcher (not just those at heavily-resourced institutions) to build AI systems which leverage large-scale synthetic indoor training data.The planned dataset construction pipeline will construct 3D scenes based on 2D floor plan datasets, which already exist at large scale. Using a machine-learning-based system previously developed by the investigators, these 2D floor plans will be converted to 3D models of empty houses. Then, each room in the house will be populated with objects in a plausible arrangement. Initially, this step will be performed by crowd workers on a platform such as Amazon Mechanical Turk. The workers will be instructed to place objects so as to match a photograph, where the photograph is chosen such that its (estimated) room geometry matches the geometry of the empty room to be populated. In a later stage of the project, rooms populated in this manner will be used to train a machine learning model which can automatically place objects based on an input photograph, thus further accelerating the dataset construction process.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.