CSR: Small: Reconfigurable In-Sensor Architectures for High Speed and Low Power In-situ Image Analysis

摄像机广泛用于监视和监控应用，可以捕获大量的图像数据。然而，这些数据的处理要么在后验执行，要么在功能强大的后端服务器上执行。虽然后验和非实时视频分析对于某些应用来说可能已经足够了，但对于复杂环境中的自主导航或无人机上使用相机的高光谱图像分析等应用来说，它还不够，因为这些应用需要近实时的视频和图像分析，有时还受到SWAP（尺寸重量和功率）的限制。这项工作假设，通过将计算推进到图像传感器中，可以克服实时成像中未来的数据挑战。这种系统将利用传感器阵列的大规模并行特性来减少处理单元分析的数据量。为此，垂直集成技术，如焦平面传感器处理器（FPSP），已经发展克服传统的图像处理系统的局限性。虽然这些设备中的一些是可编程的，并提供近距离传感器处理的好处，如性能和带宽减少，但它们显示出许多缺点。例如，每个像素列都由单个处理器处理，这减少了并行性，所有像素都被平等对待并以相同的速率处理，尽管当前应用程序的输入相关性存在差异。因此，系统花更多的时间在不相关的数据上，这增加了感知和计算时间和功耗。对fpsp的研究主要集中在技术方面，有一些概念的证明。体系结构设计方法涉及到将应用程序映射到低层体系结构的高级综合，但没有得到很多关注。为了克服现有架构的局限性，本研究的目标是设计一个高度并行、分层、可重构和垂直集成的3D传感计算架构（XPU），以及用于实时、低功耗视频分析的高级合成方法。该体系结构由分层交织的平面组成，每个平面由称为xpu的计算单元组成。底层平面并行处理像素以确定图像中的底层形状，而高层平面使用底层平面的输出来推断图像中的全局特征。提出的架构提出了三个新的贡献：用于视频流并行特征提取的分层可配置架构，基于机器学习的关联反馈方法，该方法适应计算性能和资源使用以输入数据相关性，以及将顺序图像处理算法转换为传感器中多层并行计算处理单元的框架。这些项目的成果可用于其他领域，在这些领域，需要对部署在现场的通用传感器收集的数据进行大量处理。此外，将顺序结构转换为使用硬件结构的功能等效加速器的机制将导致高度并行和高效的传感单元，可以更有效地执行特定领域的任务。

Cameras are pervasively used for surveillance and monitoring applications and can capture a substantial amount of image data. The processing of this data, however, is either performed a posteriori or at powerful backend servers. While a posteriori and non-real-time video analysis may be sufficient for certain groups of applications, it does not suffice for applications such as autonomous navigation in complex environments, or hyper spectral image analysis using cameras on drones, that require near real-time video and image analysis, sometimes under SWAP (Size Weight and Power) constraints. This work hypothesizes that future data challenges in real-time imaging can be overcome by pushing computation into the image sensor. Such systems will exploit the massive parallel nature of sensor arrays to reduce the amount of data analyzed at the processing unit. To this end, vertically integrated technology, such as focal plane sensor processors (FPSP), have been developed to overcome the limitations of conventional image processing systems. While some of these devices are programmable and offer the benefits of close-to-sensor processing such as performance and bandwidth reduction, they exhibit many drawbacks. For instance, each column of pixels is handled by a single processor, which reduces the parallelism and all pixels are treated equally and processed at the same rate, despite differences in input relevance for the application at hand. Consequently, systems spend more time spinning on non-relevant data, which increases sensing and computation time and power consumption. Research on FPSPs has mostly focused on technology aspects with some proof of concepts. Architectural design approaches, that involve high-level synthesis with the goal of mapping applications to low-level architectures, have not gained a lot of attention.To overcome the limitations of existing architectures, the goal of this research is the design of a highly parallel, hierarchical, reconfigurable and vertically-integrated 3D sensing-computing architecture (XPU), along with high-level synthesis methods for real-time, low-power video analysis. The architecture is composed of hierarchical intertwined planes, each of which consists of computational units called XPUs. The lowest-level plane processes pixels in parallel to determine low level shapes in an image while higher-level planes use outputs from low-level planes to infer global features in the image. The proposed architecture presents three novel contributions: a hierarchical, configurable architecture for parallel feature extraction in video streams, a machine learning based relevance-feedback method that adapts computational performance and resource usage to input data relevance, and a framework for converting sequential image processing algorithms to multiple layers of parallel computational processing units in the sensor. The results of this projects can be used in other fields, where large amounts of processing need to be performed on data collected by generic sensors deployed in the field. Furthermore, mechanisms for translating sequential constructs into functionally equivalent accelerators using hardware constructs will lead to highly parallel and efficient sensing units that can perform domain specific tasks more efficiently.