Modelling Image Formation and In-Camera Imaging Pipelines

我的研究项目专注于通过数码相机拍摄的图像来理解物理世界。更具体地说，我感兴趣的是一张图像能告诉我们关于现实世界环境的什么。为此，我的研究重点是如何对进入相机的物理光如何转换为最终的三通道图像(红、绿、蓝)基于像素的图像进行建模。这种类型的图像形成有时被称为“低级计算机视觉”，它处理图像是如何形成的以及它们与物理世界的关系。低级计算机视觉通常将图像视为2D信号，很像通信信号。这样，我们就可以讨论一些问题，例如当真实信号经历了一些退化(例如传感器噪声或图像处理过程中的运动，例如由于相机运动导致的图像模糊)时尝试确定真实信号。这样的成像模型还可以考虑环境因素，如雾和雾霾，以及如何通过计算方法消除这些因素。 作为这个研究计划的一部分，我的工作的一个关键组成部分是准确地了解数码相机的工作原理。虽然我们喜欢将相机视为测光设备，但目前商用相机的设计包括在相机上应用的大量额外处理(通常称为相机内处理管道)。大多数相机制造商的目标是制作视觉上令人愉快的照片，而不一定是忠实地捕捉拍摄到的场景。虽然这对于摄影来说是理想的，但这种类型的图像处理经常与低级计算机视觉中使用的模型不一致。特别是，相机内的操作可能会修改颜色，改变局部对比度，并以一种使确定物理环境的实际性质具有挑战性的方式严重扭曲原始传感器的响应。我的研究重点之一是设计新的相机处理流水线，使其能够产生摄影图像和适合科学目的的图像。开发这种“混合相机”可能会产生重大影响，因为我们现在正在使用我们的相机(特别是移动设备上的相机)来执行许多非以照片为中心的任务(如文件扫描、物体识别、医学成像、色彩匹配)。这项研究计划的长期前景是塑造未来消费相机的设计以及它们可以用于的应用。

My research program is focused on understanding the physical world through images captured by digital cameras. More specifically, I'm interested in what an image can tell us about the real-world environment. To this end, my research focuses on ways to model how physical light coming into the camera is transformed into the final three-channel image (red, green, blue) pixel-based images. This type of image formation is sometimes referred to as "low-level computer vision", which deals with how images are formed and their relationship to the physical world. Low-level computer vision often treats an image as a 2D signal, much like a communication signal. In this way, we can discuss problems such as trying to determine the true signal when it has undergone some degradation (such as sensor noise, or movement during the image process e.g. image blur due to camera motion). Such image formation models can also take into consideration environment factors, such as fog and haze, and how this can be removed through computational methods. As part of this research program, one of the key components of my work is understanding exactly how digital cameras work. While we like to think of cameras as light-measuring devices, the current design of commodity cameras includes a great deal of additional processing that is applied on board the camera (often referred to as the in-camera processing pipeline). The goal of most camera manufacturers is to make visually pleasing photographs and not necessarily to faithfully capture the imaged scene. While this is ideal for photography, this type of image manipulation is often at odds with models used in low-level computer vision. In particular, in-camera manipulation can modify colours, change local contrast, and substantially distort the original sensor response in a way that makes it challenging to determine the actual nature of the physical environment. One of my research focuses is to design new camera processing pipelines that allow the ability to produce both photographic images and images suitable for scientific purposes. Developing such "hybrid cameras" has the potential for significant impact, as we now are using our cameras (especially those on our mobile devices) for many non-photo-centric tasks (such as document scanning, object identification, medical imaging, colour matching). The long-term prospects of this research program are to shape the future design of consumer cameras and the applications they can be used for.