CCRI: Planning: A Community-Standard, Large-Scale Synthetic 3D Scene Dataset for Scene Analysis and Synthesis

为了成为有用的家庭助手，机器人需要了解它们所看到的东西以及如何在室内环境中导航。目前解决这些问题的最先进方法依赖于机器学习，特别是深度学习，这需要大量的标记数据（例如，许多图像具有每像素标签，指示该像素处存在什么类型的对象）。与其要求人们费力地标记从真实世界空间捕获的数据，一种有前途的替代方法是使用 * 合成 * 3D场景：室内空间的虚拟3D模型。填充这些虚拟空间的3D对象可以配备诸如其对象类型之类的信息，这允许基本上“免费”创建大量标记的训练数据。该项目旨在构建 * 社区标准的大规模合成3D场景数据集。虽然存在一些合成3D场景数据集，但它们要么太小，要么由于其3D模型的版权问题而受到繁重的使用限制（甚至诉讼），这些数据集通常来自营利性公司。这个项目将构建一个大规模的数据集的免费提供的3D内容。该项目的主要贡献不仅仅是这个数据集，还包括用于创建此类3D场景数据集的可扩展管道。该管道将作为开源发布，允许其他人扩展数据集或构建自己的数据集，以满足今天可能难以预测的需求。总的来说，该项目的成果将使任何研究人员（不仅仅是那些资源丰富的机构）都能够构建利用大规模合成室内训练数据的人工智能系统。计划中的数据集构建管道将基于已经存在的大规模2D平面图数据集构建3D场景。使用研究人员先前开发的基于机器学习的系统，这些2D平面图将被转换为空房子的3D模型。然后，房子中的每个房间将以合理的排列填充对象。最初，此步骤将由群组工作人员在Amazon Mechanical Turk等平台上执行。工作人员将被指示放置对象以匹配照片，其中选择照片以使其（估计）房间几何形状匹配待填充的空房间的几何形状。在项目的后期阶段，以这种方式填充的房间将用于训练机器学习模型，该模型可以根据输入照片自动放置对象，从而进一步加快数据集构建过程。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

To function as useful household assistants, robots need to understand what they are seeing and how to navigate in indoor environments. The current state-of-the-art approaches for solving these problems rely on machine learning, and in particular deep learning, which requires large quantities of labeled data (e.g. many images with per-pixel labels indicating what type of object is present at that pixel). Rather than asking people to laboriously label data captured from real-world spaces, a promising alternative approach is to use *synthetic* 3D scenes: virtual 3D models of indoor spaces. The 3D objects which populate these virtual spaces can be equipped with information such as their object type, which allows large sets of labeled training data to be created essentially “for free.” This project aims to construct *the* community-standard, large-scale synthetic 3D scene dataset. While some synthetic 3D scene datasets exist, they are either too small, or they have been subject to onerous use restrictions (and even lawsuits) due to copyright issues on their 3D models, which typically come from for-profit companies. This project will construct a large-scale dataset out of freely-available 3D content. The main contribution of the project is not just this dataset, but also a *scalable pipeline* for creating such 3D scene datasets. This pipeline will be released as open source, allowing others to expand the dataset or to construct their own datasets for needs which may be difficult to anticipate today. In total, the results of this project will enable any researcher (not just those at heavily-resourced institutions) to build AI systems which leverage large-scale synthetic indoor training data.The planned dataset construction pipeline will construct 3D scenes based on 2D floor plan datasets, which already exist at large scale. Using a machine-learning-based system previously developed by the investigators, these 2D floor plans will be converted to 3D models of empty houses. Then, each room in the house will be populated with objects in a plausible arrangement. Initially, this step will be performed by crowd workers on a platform such as Amazon Mechanical Turk. The workers will be instructed to place objects so as to match a photograph, where the photograph is chosen such that its (estimated) room geometry matches the geometry of the empty room to be populated. In a later stage of the project, rooms populated in this manner will be used to train a machine learning model which can automatically place objects based on an input photograph, thus further accelerating the dataset construction process.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.