Mask-ToF: Learning Microlens Masks for Flying Pixel Correction in Time-of-Flight Imaging
Mask-ToF: Learning Microlens Masks for Flying Pixel Correction in Time-of-Flight Imaging
复制标题
DOI：
10.1109/cvpr46437.2021.00900
复制
发表时间：
2021-03
期刊：
影响因子：
--
通讯作者：
Ilya Chugunov;Seung-Hwan Baek;Q. Fu;W. Heidrich;Felix Heide
中科院分区：
文献类型：
--
作者：
Ilya Chugunov;Seung-Hwan Baek;Q. Fu;W. Heidrich;Felix Heide
我们介绍了一种在飞行时间(ToF)深度捕获中减少飞行像素(FP)的方法MASK-ToF。FPS是发生在深度边缘周围的普遍伪影，其中来自对象及其背景的光路在光圈上积分。这种光混合在传感器像素上会产生错误的深度估计，这可能会对下游的3D视觉任务产生不利影响。MASK-ToF从这些FFP的源头开始，学习微透镜级别的遮挡掩模，该掩模有效地为每个传感器像素创建自定义形状子孔径。这在每个像素的基础上调制前景和背景光混合的选择，从而将场景几何信息直接编码到ToF测量中。我们开发了一个可微分的ToF模拟器来联合训练卷积神经网络来解码这些信息，并产生高保真、低FP深度的重建。我们在模拟光场数据集上测试了MASK-ToF的有效性，并用一个实验原型验证了该方法。为此，我们制作了学习的幅度掩模，并设计了一个光学中继系统，将其虚拟地放置在高分辨率的ToF传感器上。我们发现，MASK-ToF在不需要重新训练的情况下很好地推广到真实数据，将FP计数减少了一半。
We introduce Mask-ToF, a method to reduce flying pixels (FP) in time-of-flight (ToF) depth captures. FPs are pervasive artifacts which occur around depth edges, where light paths from both an object and its background are integrated over the aperture. This light mixes at a sensor pixel to produce erroneous depth estimates, which can adversely affect downstream 3D vision tasks. Mask-ToF starts at the source of these FPs, learning a microlens-level occlusion mask which effectively creates a custom-shaped sub-aperture for each sensor pixel. This modulates the selection of foreground and background light mixtures on a per-pixel basis and thereby encodes scene geometric information directly into the ToF measurements. We develop a differentiable ToF simulator to jointly train a convolutional neural network to decode this information and produce high-fidelity, low-FP depth reconstructions. We test the effectiveness of Mask-ToF on a simulated light field dataset and validate the method with an experimental prototype. To this end, we manufacture the learned amplitude mask and design an optical relay system to virtually place it on a high-resolution ToF sensor. We find that Mask-ToF generalizes well to real data without retraining, cutting FP counts in half.