SHF: Small: Collaborative Research: Software-Defined Imaging for Energy-Efficient Visual Computing

随着自动导航、健康监测、机器人等未来智能系统的发展，图像传感器在日常生活中变得无处不在。在这些相机驱动的应用中，一个核心挑战是当前传感器设计的不灵活性及其随之而来的能源成本。本项目设计了一种新型的图像传感器，它利用硬件-软件协同设计以更低的成本获得更好的传感。该项目推进了从混合信号传感器电路到计算架构和操作系统软件支持的垂直集成设计。该项目的影响是创造新的、灵活的图像传感器系统，可用于各种视觉计算应用。该项目进一步寻求通过强调未来智能相机的课程材料来扩大教育和研究的机会，在暑期项目中向初高中学生推广成像应用，并通过软件定义成像研讨会参与行业。该项目侧重于设计软件定义的图像传感器，它提供了可配置性的新维度，以及系统支持和编程抽象，以支持特定应用的需求。为实现这一目标，该项目侧重于三个主要目标：(1)可配置传感器原语的设计和实现，包括具有自定义曝光、读出和量化的可编程区域；(2)控制这些传感器原语的架构，并加速图像信号处理和视觉工作负载；(3)为我们的新传感器原语调度内存需求的操作系统服务，以及为关键应用实现低延迟反应传感器控制的操作系统接口。这些创新在一个综合评估试验台中进行评估，该试验台包括一个制造的软件定义图像传感器原型和一个基于现场可编程门阵列（FPGA）的系统，用于测量一组端到端视觉应用的能量和性能。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Image sensors are becoming ubiquitous in daily life as they are incorporated in future intelligent systems including autonomous navigation, health monitoring, and robotics. A central challenge in these camera-driven applications is the inflexibility of current sensor designs and their consequent energy cost. This project designs a new category of image sensors which exploit hardware -- software co-design to attain better sensing at lower cost. The project advances a vertically-integrated design from the mixed-signal sensor circuitry to the computational architecture and the operating system software support. The project's impacts are the creation of new, flexible image sensor systems that can be used for a variety of visual computing applications. The project further seeks to include broadening access to education and research through curriculum material which emphasize smart cameras of the future, outreach to middle and high school students in a summer program to discover imaging applications, and industry engagement through workshops on software-defined imaging. The project focuses on designing software-defined image sensors, which offer new dimensions of configurability along with system support and programming abstractions to support application-specific needs. To achieve this, the project focuses on three main objectives: (1) Design and implementation of configurable sensor primitives, including programmable regions of interest with custom exposure, readout, and quantization, (2) architectures to control these sensor primitives, and to accelerate image signal processing and vision workloads, and (3) operating system services for scheduling the memory needs for our new sensor primitives, and OS interfaces that enable low-latency reactive sensor control for key applications. These innovations are evaluated in an integrative evaluation testbed that includes a fabricated software-defined image sensor prototype and a Field Programmable Gate Array (FPGA)-based system to measure energy and performance for a set of end-to-end visual applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.