VEC: Medium: The 3D Memex for Virtual Teleportation

尽管虚拟和增强现实头戴设备最终处于大规模消费者可用性的边缘，但人们对如何以最令人兴奋的VR和AR应用所需的保真度捕捉世界知之甚少：设计一个家庭改造计划，通过添加固定装置或拆除墙壁与一个手指轻弹，传送到夏威夷“走过”度假租赁，或者在沙发上和一个近亲交谈，就好像你们真的在同一个房间里一样。实现这种远距离传送场景需要超越当前技术水平的3D场景建模和渲染。阻碍3D前沿进展的主要因素是缺乏数据。 网上有数十亿张照片，这已经彻底改变了图像理解。 由于公开的3D数据非常稀缺，本项目采用数据驱动的方法来研究虚拟隐形传态问题。 它的核心是创建一个3D Memex：一个巨大的，不断发展的，免费提供的3D几何，分割和语义标签的在线存储库。 3D Memex将捕捉广泛的家居物品和场景。 除了几何和标签，3D Memex将描述关节和自由度。这项研究将利用这个新的数据库，产生3D-Memex感知的扫描和建模算法，使最先进的技术更接近高质量的沉浸式VR体验。 Memex感知的重建算法将推断出完整的场景模型，通过匹配3D Memex中的类似场景并完成隐藏的几何和纹理，推断出远远超出相机所看到的内容。 它将构建完整的模型和运动自由度（即，门会打开和关闭，椅子会移动，等等） 这些突破性的能力将使重要的远距离传送方案在项目中原型化。远距离传送能力将为所有人带来新的和更好的应用，包括真实的房地产、旅游业，以及通过实时远程呈现实现更好的人际交流。 所有数据沿着可视化和标记工具将与新开发的下一代3D扫描工具沿着免费在线发布给更广泛的社区。 最后，该项目将通过让本科生直接参与正在进行的研究来推进教育目标，并通过为本科生和研究生开设的顶点课程来帮助播种3D Memex，并让学生接触快速发展的虚拟现实技术和算法。

Although virtual and augmented reality headsets are finally on the brink of mass consumer availability, very little is known about how to capture the world with the fidelity needed to enable the most exciting applications of VR and AR: designing a home remodeling plan by adding fixtures or removing walls with a flick of a finger, teleporting to Hawaii to "walk through" a vacation rental, or having a conversation with a close relative on their couch, as if you're literally in the same room. Achieving such teleportation scenarios will require 3D scene modeling and rendering that go beyond the current state of the art. The primary factor holding back progress on the 3D front is lack of data. There are billions of photos available online, and this has revolutionized image understanding. Since publicly available 3D data is comparably scarce, this project takes a data-driven approach to the research problem of virtual teleportation. Its centerpiece is the creation of a 3D Memex: a vast, evolving, freely available, online repository of 3D geometry, segmentations, and semantic labels. The 3D Memex will capture a broad range of household objects and scenes. Beyond geometry and labels, the 3D Memex will describe articulations and degrees of freedom.This research will leverage this new database, producing 3D-Memex-aware scanning and modeling algorithms to bring the state of the art closer to high quality immersive VR experiences. Memex-aware reconstruction algorithms will infer complete scene models, extrapolating far beyond what the camera sees, by matching to similar scenes in the 3D Memex and completing hidden geometry and texture. Instead of reconstructing static point clouds, it will build complete models and kinematic degrees of freedom (i.e., doors will open and close, chairs move, etc.) These breakthrough capabilities will enable important teleportation scenarios to be prototyped in the project. The teleportation capabilities will enable new and better applications for all, including real estate, tourism, and better interpersonal communication through live telepresence. All data, along with visualization and labeling tools, will be released freely online for the broader community, along with newly developed, next-generation 3D scanning tools. Finally, the project will advance educational objectives by engaging undergraduates directly in the ongoing research, and through a capstone course for undergraduates and graduates that will help seed the 3D Memex and expose the students to the rapidly growing technologies and algorithms for virtual reality.