Differentiable Compound Optics and Processing Pipeline Optimization for End-to-end Camera Design
Differentiable Compound Optics and Processing Pipeline Optimization for End-to-end Camera Design
复制标题
DOI：
10.1145/3446791
复制
发表时间：
2021-06-01
影响因子：
6.2
通讯作者：
Heide, Felix
中科院分区：
文献类型：
--
作者：
Tseng, Ethan;Mosleh, Ali;Heide, Felix
我们直接用于成像或间接依赖于下游应用的大多数现代商品成像系统采用多个镜头的光学系统，必须平衡与完美光学的偏差、制造限制、公差、成本和占地面积。虽然光学设计通常与下游图像处理或分析任务有复杂的相互作用，但今天的复合光学设计与这些相互作用无关。现有的光学设计工具旨在最小化光学像差，例如与高斯光学线性模型的偏差，而不是特定于应用的损失，从而排除了与硬件图像信号处理（ISP）和高度参数化的神经网络处理的联合优化。在这篇文章中，我们提出了一种复合光学的优化方法，解除这些限制。我们优化整个透镜系统，包括硬件和软件图像处理管道、下游神经网络处理以及特定于应用的端到端损耗。为此，我们提出了一种用于复合光学的可学习的、可微分的前向模型和一种交替的邻近优化方法，该方法可以处理光学、硬件ISP和神经网络的参数维度高度变化的函数组合。我们的方法与现有的光学设计工具（如ZEMAX）无缝集成。因此，我们可以在许多相机系统设计和端到端应用中评估我们的方法。我们验证了我们的方法，在汽车摄像头的光学设置与硬件ISP后处理和检测优于经典的光学设计，汽车物体检测和交通灯状态检测。对于人类观看任务，我们优化了动态户外场景和动态低光成像的光学和处理管道。在所有测试的特定领域应用程序中，我们在定性和定量上都优于现有的分区设计或微调方法。
Most modern commodity imaging systems we use directly for photography-or indirectly rely on for downstream applications-employ optical systems of multiple lenses that must balance deviations from perfect optics, manufacturing constraints, tolerances, cost, and footprint. Although optical designs often have complex interactions with downstream image processing or analysis tasks, today's compound optics are designed in isolation from these interactions. Existing optical design tools aim to minimize optical aberrations, such as deviations from Gauss' linear model of optics, instead of application-specific losses, precluding joint optimization with hardware image signal processing (ISP) and highly parameterized neural network processing. In this article, we propose an optimization method for compound optics that lifts these limitations. We optimize entire lens systems jointly with hardware and software image processing pipelines, downstream neural network processing, and application-specific end-to-end losses. To this end, we propose a learned, differentiable forward model for compound optics and an alternating proximal optimization method that handles function compositions with highly varying parameter dimensions for optics, hardware ISP, and neural nets. Our method integrates seamlessly atop existing optical design tools, such as ZEMAX. We can thus assess our method across many camera system designs and end-to-end applications. We validate our approach in an automotive camera optics setting-together with hardware ISP post processing and detection-outperforming classical optics designs for automotive object detection and traffic light state detection. For human viewing tasks, we optimize optics and processing pipelines for dynamic outdoor scenarios and dynamic low-light imaging. We outperform existing compartmentalized design or fine-tuning methods qualitatively and quantitatively, across all domain-specific applications tested.