Modelling Image Formation and In-Camera Imaging Pipelines

我的研究项目专注于通过数码相机捕捉的图像来理解物理世界。更具体地说，我感兴趣的是图像可以告诉我们关于现实世界的环境。为此，我的研究重点是如何建模进入相机的物理光如何转换为最终的三通道图像（红，绿色，蓝色）基于像素的图像。这种类型的图像形成有时被称为“低级计算机视觉”，它处理图像如何形成以及它们与物理世界的关系。低级计算机视觉通常将图像视为2D信号，就像通信信号一样。通过这种方式，我们可以讨论一些问题，例如当信号经历了一些退化（例如传感器噪声或图像处理期间的移动，例如由于相机运动导致的图像模糊）时，试图确定真实信号。这种图像形成模型还可以考虑环境因素，例如雾和霾，以及如何通过计算方法去除这些因素。 作为这项研究计划的一部分，我工作的关键组成部分之一是了解数码相机的工作原理。虽然我们喜欢将相机视为光测量设备，但目前商品相机的设计包括大量应用于相机上的附加处理（通常称为相机内处理管道）。大多数相机制造商的目标是制作视觉上令人愉悦的照片，而不一定要忠实地捕捉成像场景。虽然这是摄影的理想选择，但这种类型的图像处理通常与低级计算机视觉中使用的模型不一致。特别是，相机内操纵可以修改颜色，改变局部对比度，并以一种使确定物理环境的实际性质具有挑战性的方式大幅扭曲原始传感器响应。我的研究重点之一是设计新的相机处理管道，使其能够生成摄影图像和适合科学目的的图像。开发这样的“混合相机”有可能产生重大影响，因为我们现在正在使用我们的相机（特别是我们的移动的设备上的相机）进行许多非以照片为中心的任务（例如文档扫描，对象识别，医学成像，颜色匹配）。这项研究计划的长期前景是塑造消费相机的未来设计及其可用于的应用。

My research program is focused on understanding the physical world through images captured by digital cameras. More specifically, I'm interested in what an image can tell us about the real-world environment. To this end, my research focuses on ways to model how physical light coming into the camera is transformed into the final three-channel image (red, green, blue) pixel-based images. This type of image formation is sometimes referred to as "low-level computer vision", which deals with how images are formed and their relationship to the physical world. Low-level computer vision often treats an image as a 2D signal, much like a communication signal. In this way, we can discuss problems such as trying to determine the true signal when it has undergone some degradation (such as sensor noise, or movement during the image process e.g. image blur due to camera motion). Such image formation models can also take into consideration environment factors, such as fog and haze, and how this can be removed through computational methods. As part of this research program, one of the key components of my work is understanding exactly how digital cameras work. While we like to think of cameras as light-measuring devices, the current design of commodity cameras includes a great deal of additional processing that is applied on board the camera (often referred to as the in-camera processing pipeline). The goal of most camera manufacturers is to make visually pleasing photographs and not necessarily to faithfully capture the imaged scene. While this is ideal for photography, this type of image manipulation is often at odds with models used in low-level computer vision. In particular, in-camera manipulation can modify colours, change local contrast, and substantially distort the original sensor response in a way that makes it challenging to determine the actual nature of the physical environment. One of my research focuses is to design new camera processing pipelines that allow the ability to produce both photographic images and images suitable for scientific purposes. Developing such "hybrid cameras" has the potential for significant impact, as we now are using our cameras (especially those on our mobile devices) for many non-photo-centric tasks (such as document scanning, object identification, medical imaging, colour matching). The long-term prospects of this research program are to shape the future design of consumer cameras and the applications they can be used for.