Embedded vision systems - a platform for integrating modern imaging sensors and real-time image processing

百万像素图像传感器在我们的现代社会中很常见，可以在手机、游戏机、静止图像和摄像机等设备中找到。在这一巨大细分市场运营的传感器制造商之间的激烈竞争压低了成本，同时使具有新功能的传感器的开发成为可能并加快了速度。传感器技术的这场革命使波长有限的像素大小的超大型阵列在消费产品中可用。与此同时，诸如低灵敏度和高噪声水平等问题已经被成功地解决，以至于传感器成为要求更高的科学和工业社区对非常多样化的应用的兴趣。由此产生的高信噪比和新兴传感器的高帧速率能力使需要高动态范围(HDR)的重要成像任务成为可能。另一个令人兴奋的发展是，通过测量入射到每个像素上的光(光谱)的颜色，为2D成像增加了进一步的维度。曝光期间传感器的非破坏性读出允许在传感器的动态范围内获得非常大的增益，尽管代价是必须为单个HDR图像多次读出传感器。3D、高分辨率、高动态范围和高帧速率的组合正在加剧一个已经很严重的问题，即以非常低的帧速率(每秒30帧)处理和传输数据。许多现有的专业摄像机完全填满了最高速度数据传输协议提供的带宽。撇开带宽问题不谈，图像处理的任务仍然是：简单地用数据填充大内存缓冲区以进行离线处理并不适合许多应用程序。为了解决我们由EPSRC资助的高能粒子束透明成像和诊断工作中数据雪崩的严重问题，我们从大自然中获得了灵感。例如，在人类视觉中，来自光敏感细胞的信息不会立即逐个像素地进入大脑，但已经在视网膜中，通过组合来自几个视觉感受器的信号来消化一定数量的数据。复杂的前端图像处理方法是在高帧速率下测量关键高分辨率光束特性的唯一方法。经过三代相机的设计和10人年的努力，我们已经改进了一种新颖的相机架构，它结合了粗粒度和细粒度的并行图像处理，使用了不同层次的图像处理元素。有了这些，只传输那些逐帧变化的像素就成了标准。我们已经实现了各种专用的实时图像处理任务，例如通过只传递图像中像素强度和/或颜色在帧之间变化的那部分来减少数据流、HDR成像、粒子跟踪和尺寸测量、颜色内插、降噪、特征提取、组织图和许多其他任务。我们在这个项目中提出的工作将使我们能够向市场展示我们的系统的能力，并提供对该系统表现出兴趣的选定小组。我们将展示我们的方法将如何变革许多复杂的成像技术，就像数码相机改变了摄影一样。我们将把我们的设备转变为“即插即用”的摄像头系统，以便其他人随时使用。该项目汇集了学术和商业背景的专家，为我们的视觉系统技术上市做好准备，并探索将创造的知识产权和专有技术进行商业利用的最佳途径。合作伙伴将共同实施两项关键应用：光谱成像和将该设备整合为曼彻斯特无人驾驶车辆计划的大脑和眼睛。项目完成后，我们将处于有利地位，可以获得风险资本或种子基金的支持。

Megapixel image sensors are commonplace in our modern society and are found in devices such as mobile phones, game consoles, still and video cameras. The fierce competition between sensor manufacturers that operate in this large market segment has driven down costs and at the same time enabled and accelerated the development of sensors with novel capabilities. This revolution in sensor technology has made very large arrays with wavelength limited pixel size available in consumer products. At the same time problematic issues such as low sensitivity and high noise levels have been addressed successfully to the point that the sensors become of interest to the more demanding scientific and industrial community for a very diverse range of applications. The resulting high signal to noise ratio and high frame rate capability of emerging sensors enable important imaging tasks that require high dynamic range (HDR). A further exciting development is that of adding a further dimension to the 2D imaging by measuring the 'colour' of the light (spectroscopic) incident on each pixel. A nondestructive readout of the sensor during exposure allows very large gains in the dynamic range of a sensor albeit at the expense of having to read out the sensor multiple times for a single HDR image. The combination of 3D, high resolution, high dynamic range and high frame rate is exacerbating an already serious issue for handling and transmitting data at very modest frame rates of 30 per second. Many existing professional cameras completely fill the bandwidth that the highest speed data transfer protocols offer. Bandwidth issues aside, the task of image processing remains: simply filling large memory buffers with data for offline processing doesn't suit many applications.To address this serious issue of data avalanche in our EPSRC funded work on transparent imaging and diagnostics of highly energetic particle beams, we took inspiration from nature. In human vision, for example, the information from the light sensitive cells does not immediately go to the brain, pixel by pixel, but already in the retina a certain amount of data is digested by combining the signals from several visual receptors. A sophisticated front-end image processing approach was the only way to measure the key high-resolution beam characteristics at high frame rates. After three generations of camera designs and 10 man years of effort, we have refined a novel camera architecture that combines both coarse and fine grain parallel image processing using a heterogeneous hierarchy of image processing elements. With these it becomes standard to only transmit those pixels that vary frame by frame. We have already implemented various dedicated realtime image processing tasks such as reducing data flow by only passing on that part of the image that changes between frames in pixel intensity and/or colour, HDR imaging, particle tracking and size measurement, colour interpolation, noise reduction, feature extraction, histogramming, and many others. The work we propose in this project will allow us to show the market what our systems are capable of and to provide selected groups that show an interest with the systems. We will demonstrate how our approach will revolutionize many sophisticated imaging techniques in much the same way as digital still cameras have transformed photography. We will turn our devices to 'plug and play' camera systems that can readily be used by others. This project brings together specialists from both academic and commercial background to prepare our vision systems technology for market and to investigate the best route to commercial exploitation of the created IP and know-how. Together the partners will implement two key applications: spectroscopic imaging and integrate the device as the brain and eyes of an unmanned vehicle initiative at Manchester. On completion of the project we will be in a strong position to secure support from venture capital or seed funds.