MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images
MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images
复制标题
DOI：
10.1007/978-3-030-58452-8_26
复制
发表时间：
2020-08
期刊：
影响因子：
--
通讯作者：
Benjamin Attal;Selena Ling;Aaron Gokaslan;Christian Richardt;J. Tompkin
中科院分区：
文献类型：
--
作者：
Benjamin Attal;Selena Ling;Aaron Gokaslan;Christian Richardt;J. Tompkin
我们介绍了一种将立体360(全方位立体)图像转换为分层的多球体图像表示的方法，用于六自由度(6DoF)渲染。立体360图像可以从虚拟现实(VR)的多摄像机系统中捕获，但缺乏运动视差和全方位校正的视差提示。这些因素加在一起，很快就会导致观看内容时出现VR病。一种解决方案是尝试生成一种适合6DoF渲染的格式，例如通过估计深度。然而，这引发了如何在动态场景中处理不遮挡区域的问题。我们的方法是通过多球图像表示同时学习深度和错位，该图像表示可以在VR中使用正确的6DoF视差和运动视差来渲染。这显著改善了观众的舒适度，并且可以在现代GPU硬件上实时推断和渲染。总而言之，这些都将使VR视频成为一种更舒适的身临其境的媒介。
We introduce a method to convert stereo 360(omnidirectional stereo) imagery into a layered, multi-sphere image representation for six degree-of-freedom (6DoF) rendering. Stereo 360imagery can be captured from multi-camera systems for virtual reality (VR), but lacks motion parallax and correct-in-all-directions disparity cues. Together, these can quickly lead to VR sickness when viewing content. One solution is to try and generate a format suitable for 6DoF rendering, such as by estimating depth. However, this raises questions as to how to handle disoccluded regions in dynamic scenes. Our approach is to simultaneously learn depth and disocclusions via a multi-sphere image representation, which can be rendered with correct 6DoF disparity and motion parallax in VR. This significantly improves comfort for the viewer, and can be inferred and rendered in real time on modern GPU hardware. Together, these move towards making VR video a more comfortable immersive medium.