VEC: Medium: The 3D Memex for Virtual Teleportation

虽然虚拟和现实增强耳机终于在大众消费的边缘可用性,完全不知道如何捕捉世界所需的忠诚使最令人激动的应用虚拟现实和基于“增大化现实”技术:设计一个家庭装修计划通过增加设备或删除墙手指轻轻一推,传送到夏威夷度假租赁“走过”,与近亲属或有一个对话在沙发上,好像你在同一个房间里。实现这样的隐形传输场景将需要3D场景建模和渲染，超越目前的艺术状态。阻碍3D技术发展的主要因素是缺乏数据。网上有数十亿的照片，这给图像理解带来了革命性的变化。由于公开可用的3D数据相对较少，该项目采用数据驱动的方法来研究虚拟传送问题。它的核心是3D Memex的创建：一个巨大的、不断发展的、免费的3D几何、分割和语义标签的在线存储库。3D Memex将捕捉广泛的家庭物品和场景。除了几何形状和标签，3D Memex将描述关节和自由度。这项研究将利用这个新的数据库，产生3d memex感知扫描和建模算法，使最先进的技术更接近高质量的沉浸式VR体验。Memex感知重建算法将推断出完整的场景模型，通过匹配3D Memex中的类似场景并完成隐藏的几何和纹理，推断出远远超出相机所看到的内容。而不是重建静态点云，它将建立完整的模型和运动学自由度（即，门将打开和关闭，椅子移动等），这些突破性的能力将使重要的隐形传输场景在项目中原型化。远程传送能力将为所有领域带来新的、更好的应用，包括房地产、旅游业，以及通过实时远程呈现实现更好的人际沟通。所有数据，以及可视化和标签工具，将与新开发的下一代3D扫描工具一起免费在线发布给更广泛的社区。最后，该项目将通过让本科生直接参与正在进行的研究来推进教育目标，并通过为本科生和研究生提供的顶点课程，帮助他们播种3D Memex，并让学生接触到快速发展的虚拟现实技术和算法。

Although virtual and augmented reality headsets are finally on the brink of mass consumer availability, very little is known about how to capture the world with the fidelity needed to enable the most exciting applications of VR and AR: designing a home remodeling plan by adding fixtures or removing walls with a flick of a finger, teleporting to Hawaii to "walk through" a vacation rental, or having a conversation with a close relative on their couch, as if you're literally in the same room. Achieving such teleportation scenarios will require 3D scene modeling and rendering that go beyond the current state of the art. The primary factor holding back progress on the 3D front is lack of data. There are billions of photos available online, and this has revolutionized image understanding. Since publicly available 3D data is comparably scarce, this project takes a data-driven approach to the research problem of virtual teleportation. Its centerpiece is the creation of a 3D Memex: a vast, evolving, freely available, online repository of 3D geometry, segmentations, and semantic labels. The 3D Memex will capture a broad range of household objects and scenes. Beyond geometry and labels, the 3D Memex will describe articulations and degrees of freedom.This research will leverage this new database, producing 3D-Memex-aware scanning and modeling algorithms to bring the state of the art closer to high quality immersive VR experiences. Memex-aware reconstruction algorithms will infer complete scene models, extrapolating far beyond what the camera sees, by matching to similar scenes in the 3D Memex and completing hidden geometry and texture. Instead of reconstructing static point clouds, it will build complete models and kinematic degrees of freedom (i.e., doors will open and close, chairs move, etc.) These breakthrough capabilities will enable important teleportation scenarios to be prototyped in the project. The teleportation capabilities will enable new and better applications for all, including real estate, tourism, and better interpersonal communication through live telepresence. All data, along with visualization and labeling tools, will be released freely online for the broader community, along with newly developed, next-generation 3D scanning tools. Finally, the project will advance educational objectives by engaging undergraduates directly in the ongoing research, and through a capstone course for undergraduates and graduates that will help seed the 3D Memex and expose the students to the rapidly growing technologies and algorithms for virtual reality.