A Generalized and Flexible Deep Learning Framework for the Reconstruction of High Dynamic Range Images and Video

目前的图像和视频HDR重建方法存在许多局限性，我们的目标是在本论文中解决其中的一些问题1。目前，现有技术的方法具有严格的输入要求，并且不能自适应地利用任意数量的输入图像来重建HDR图像。2.运动对齐仍然是一个具有挑战性的问题，特别是在HDR情况下，其中输入帧具有不同的曝光，并且可能面临诸如重噪声和大饱和区域的问题。3. HDR视频估计方法是计算密集型的，并且不适合诸如智能手机摄影的即时用例。为了解决上述问题，我们建议为HDR图像和视频重建开发一个通用且灵活的深度学习框架。首先，我们的方法将能够通过使用集合神经网络[15，1]的原理来使用任意数量的输入图像，从而使我们能够联合收割机结合单图像和多图像方法的优点。这将使得可以根据正在捕获的场景自适应地决定使用多少帧和哪些曝光值。当存在极端运动或图像曝光良好时，单帧可能是最佳选择，而在动态范围非常大且极端运动较少的场景中可能需要更多帧。其次，我们将研究如何针对HDR情况在帧之间执行精确的运动对齐，同时考虑到大噪声或饱和区域的问题。当前方法[6，13，11]通常应用通用运动对准模型，而不考虑与HDR重建相关联的特定挑战，因此成功有限。对这些误差源进行明确的建模将使我们能够有效地减少重影伪影并改善重建细节。最后，我们将探索将我们的方法以高效的方式扩展到视频重建的可能性，以便在智能手机上实时捕获高分辨率HDR视频内容。将于2024年2月更新（在第三年进展）

There are a number of limitations to current methods of HDR reconstruction for images and videos and we aim to address some of these in this thesis1. Currently state-of-the-art methods have rigid input requirements and cannot adaptively make use of an arbitrary number of input images to reconstruct HDR images. 2. Motion alignment remains a challenging problem, especially in the HDR case where the input frames have varying exposures and can face problems such as heavy noise and large saturated regions. 3. HDR video estimation methods are computationally intensive and unsuited to on-the-fly use cases such as smartphone photography. To address the issues outlined above, we propose to develop a generalized and flexible deep learning framework for HDR image and video reconstruction. Firstly, our method will be capable of using an arbitrary number of input images by using principles from set neural networks [15, 1], allowing us to combine the benefits of single image and multi-image approaches. This would make it possible to adaptively decide how many frames and which exposure values to use depending on the scene being captured. A single frame might be the best choice when there is extreme motion or the image is well-exposed, while more frames might be required in scenes with a very large dynamic range and less extreme motion. Secondly, we will investigate how to perform accurate motion alignment between frames specifically for the HDR case, taking into account the issues of large noisy or saturated regions. Current approaches [6, 13, 11] typically apply a generic motion alignment model without considering the specific challenges associated with HDR reconstruction and hence have limited success. Explicitly modelling these sources of error will allow us to effectively reduce ghosting artefacts and improve reconstruction detail. Finally, we will explore the possibility of extending our method to video reconstruction in a highly efficient manner for on-the-fly capture of high resolution HDR video content on smartphones. To be updated in February 2024 (at Year 3 Progression)