Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs
Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs
复制标题
DOI：
10.1109/iccv48922.2021.01604
复制
发表时间：
2021-08
期刊：
影响因子：
--
通讯作者：
Helisa Dhamo;Fabian Manhardt;Nassir Navab;F. Tombari
中科院分区：
文献类型：
--
作者：
Helisa Dhamo;Fabian Manhardt;Nassir Navab;F. Tombari
可控场景合成包括生成满足底层规范的3D信息。因此，这些规范应该是抽象的，即允许简单的用户交互，同时提供足够的接口进行详细的控制。场景图是场景的表示，由对象（节点）和对象间关系（边）组成，被证明特别适合此任务，因为它们允许对生成的内容进行语义控制。以前的工作解决这个任务往往依赖于合成数据，并检索对象网格，这自然限制了生成能力。为了避免这个问题，我们提出了第一个工作，直接从场景图中生成的形状在一个端到端的方式。此外，我们表明，相同的模型支持场景修改，使用各自的场景图作为接口。利用图卷积网络（GCN），我们在对象和边缘类别以及3D形状和场景布局之上训练变分自动编码器，允许对新场景和形状进行后期采样。
Controllable scene synthesis consists of generating 3D information that satisfy underlying specifications. Thereby, these specifications should be abstract, i.e. allowing easy user interaction, whilst providing enough interface for detailed control. Scene graphs are representations of a scene, composed of objects (nodes) and inter-object relationships (edges), proven to be particularly suited for this task, as they allow for semantic control on the generated content. Previous works tackling this task often rely on synthetic data, and retrieve object meshes, which naturally limits the generation capabilities. To circumvent this issue, we instead propose the first work that directly generates shapes from a scene graph in an end-to-end manner. In addition, we show that the same model supports scene modification, using the respective scene graph as interface. Leveraging Graph Convolutional Networks (GCN) we train a variational Auto-Encoder on top of the object and edge categories, as well as 3D shapes and scene layouts, allowing latter sampling of new scenes and shapes.