CSR: Small: Reconfigurable In-Sensor Architectures for High Speed and Low Power In-situ Image Analysis

相机广泛用于监视和监控应用，并且可以捕获大量的图像数据。然而，这些数据的处理要么是在后验中执行的，要么是在强大的后端服务器上执行的。虽然后验和非实时视频分析对于某些应用组可能是足够的，但它不足以用于诸如复杂环境中的自主导航或使用无人机上的相机的超光谱图像分析等应用，这些应用需要接近实时的视频和图像分析，有时在SWAP（尺寸重量和功率）约束下。这项工作假设，未来的实时成像数据的挑战，可以通过推动计算到图像传感器克服。这种系统将利用传感器阵列的大规模并行特性来减少处理单元分析的数据量。为此，已经开发了诸如焦平面传感器处理器（FPSP）的垂直集成技术来克服传统图像处理系统的限制。虽然这些设备中的一些是可编程的，并提供接近传感器处理的好处，如性能和带宽减少，但它们表现出许多缺点。例如，每列像素由单个处理器处理，这降低了并行性，并且所有像素都被平等地对待并以相同的速率处理，而不管手头应用的输入相关性的差异。因此，系统花费更多的时间在不相关的数据上，这增加了感测和计算时间以及功耗。FPSP的研究主要集中在技术方面，并进行了一些概念验证。为了克服现有体系结构的局限性，本研究的目标是设计一种高度并行、层次化、可重构和垂直集成的三维传感计算体系结构（XPU），沿着高级综合方法，以实现实时、可重构和可重构的三维传感计算。低功耗视频分析该架构由分层交织的平面组成，每个平面由称为XPU的计算单元组成。最低级平面并行处理像素以确定图像中的低级形状，而较高级平面使用来自低级平面的输出来推断图像中的全局特征。所提出的架构提出了三个新的贡献：一个分层的，可配置的架构，并行特征提取视频流，基于机器学习的相关性反馈方法，适应计算性能和资源使用输入数据的相关性，和一个框架，用于转换顺序图像处理算法的多层并行计算处理单元的传感器。该项目的结果可以用于其他领域，其中需要对部署在现场的通用传感器收集的数据进行大量处理。此外，用于使用硬件构造将顺序构造转换成功能上等效的加速器的机制将导致可以更有效地执行域特定任务的高度并行和高效的感测单元。

Cameras are pervasively used for surveillance and monitoring applications and can capture a substantial amount of image data. The processing of this data, however, is either performed a posteriori or at powerful backend servers. While a posteriori and non-real-time video analysis may be sufficient for certain groups of applications, it does not suffice for applications such as autonomous navigation in complex environments, or hyper spectral image analysis using cameras on drones, that require near real-time video and image analysis, sometimes under SWAP (Size Weight and Power) constraints. This work hypothesizes that future data challenges in real-time imaging can be overcome by pushing computation into the image sensor. Such systems will exploit the massive parallel nature of sensor arrays to reduce the amount of data analyzed at the processing unit. To this end, vertically integrated technology, such as focal plane sensor processors (FPSP), have been developed to overcome the limitations of conventional image processing systems. While some of these devices are programmable and offer the benefits of close-to-sensor processing such as performance and bandwidth reduction, they exhibit many drawbacks. For instance, each column of pixels is handled by a single processor, which reduces the parallelism and all pixels are treated equally and processed at the same rate, despite differences in input relevance for the application at hand. Consequently, systems spend more time spinning on non-relevant data, which increases sensing and computation time and power consumption. Research on FPSPs has mostly focused on technology aspects with some proof of concepts. Architectural design approaches, that involve high-level synthesis with the goal of mapping applications to low-level architectures, have not gained a lot of attention.To overcome the limitations of existing architectures, the goal of this research is the design of a highly parallel, hierarchical, reconfigurable and vertically-integrated 3D sensing-computing architecture (XPU), along with high-level synthesis methods for real-time, low-power video analysis. The architecture is composed of hierarchical intertwined planes, each of which consists of computational units called XPUs. The lowest-level plane processes pixels in parallel to determine low level shapes in an image while higher-level planes use outputs from low-level planes to infer global features in the image. The proposed architecture presents three novel contributions: a hierarchical, configurable architecture for parallel feature extraction in video streams, a machine learning based relevance-feedback method that adapts computational performance and resource usage to input data relevance, and a framework for converting sequential image processing algorithms to multiple layers of parallel computational processing units in the sensor. The results of this projects can be used in other fields, where large amounts of processing need to be performed on data collected by generic sensors deployed in the field. Furthermore, mechanisms for translating sequential constructs into functionally equivalent accelerators using hardware constructs will lead to highly parallel and efficient sensing units that can perform domain specific tasks more efficiently.