Episodic Semantic Scene Analysis

挑战：目标检测和识别、从传感器计算3D几何模型以及映射和定位的研究正在成熟。我们建议建立在这些能力，以创建高度可扩展的，计算效率高，3D语义表示的环境，这将使在多个应用领域，如机器人和增强现实的进步。我们解决了两个具体的挑战：1）开发一个计算效率高的语义场景映射框架的基础上的对象和相互关系，以及2）扩展这种表示，以支持有效的推理场景之间的环境变化遇到（插曲）随着时间的推移。该项目的最终目标是：1）情景语义场景分析（ESSA）的概念框架和相应的2）室内感知系统的模块化软件框架，该框架在场景结构元素和对象的水平上运行，在重建大场景（空间）和理解场景动态（时间和语义）方面具有高度可扩展性。方法：我们建议开发一个层次化的场景模型，它集成了最先进的感知算法，用于联合3D对象检测与实时几何映射和空间定位。 该模型将包含几何元素（例如平面），对象（例如椅子或桌子）和对象的星座（例如位置设置），以及它们之间的属性（例如桌子上的位置设置）。这些表示将在场景相遇时保持，允许在相遇之间快速重新接地模型，以及由于检测到的变化而有效修改模型。同时，我们的框架的语义层和几何层之间的通信可以提高推断的场景结构以及场景理解。我们将在机器人操作和增强现实的用例背景下开发ESSA。DFG的影响和预期成果：该项目的结果将有助于构建软件框架的科学和实践，这些框架支持人机交互和机器人技术中广泛的基于感知的应用。 在此过程中，

Challenge: Research on object detection and recognition, computation of 3D geometric models from sensing, and mapping and localization is maturing. We propose to build on these capabilities to create highly scalable, computationally efficient, 3D semantic representations of an environment that would enable advances in multiple application areas such a robotics and augmented reality. We address two specific challenges: 1) development of a computa-tionally efficient semantic scene mapping framework based on objects and inter-relationships, and 2) extension of this representation to support efficient reasoning about changes in the environment between scene encounters (episodes) over time. The end-goals of this project are 1) a conceptual framework for episodic semantic scene analysis (ESSA) and corresponding 2) a modular software framework for indoor perception systems that operates at the level of scene structural elements and objects, which is highly scalable in terms of both reconstructing large scenes (spatially) and understanding scene dynamics (temporally and semantically). Approach: We propose to develop a hierarchical scene model that integrates state-of-the-art perception algorithms for joint 3D object detection with real-time geometry mapping and spatial localization. The model will encompass both geometric elements (e.g. planar surfac-es), objects (e.g. chairs or tables), and constellations of objects (e.g. a place setting), and properties among them (e.g. a place setting on a table). These representations will be main-tained across scene encounters, allowing for rapid re-grounding of the model between en-counters, as well as efficient modification of the model due to detected changes. At the same time, communication between the semantic and the geometric layers of our framework can improve the inferred scene structure as well as scene understanding. We will develop ESSA on the context of use-cases from robotic manipulation and augmented reality. DFG impact & expected outcomes: The result of this project will be contributions to both the science and practice of building software frameworks that support a broad range of perception-based applications in human-computer interaction and robotics. During the course of this