SBIR Phase I: Lightweight Learning-based Camera Image Signal Processing (ISP) for Photon-Limited Imaging

这个小企业创新（SBIR）第一阶段项目的广泛影响将来自于在较低光照水平下操作数字图像传感器的能力。该技术有望部署在任何中低级相机设备中，并在利用相机技术的所有行业中具有潜在的应用。消费者应用程序将受益于改进的低照度成像，包括用于视频会议的仪表盘摄像头和笔记本摄像头；军事和国家安全应用包括夜视和自主导航；同时，该技术还将提高内窥镜检查等医疗程序的诊断能力。该技术预计将对劳动力发展产生直接影响，该解决方案的部署将推动消费电子产品的经济发展。这个小型企业创新研究（SBIR）第一阶段项目旨在使用轻量级算法实现光子限制的图像去噪，该算法有可能在相机芯片上实现。实现这一目标需要若干技术突破，共同导致一种新的图像信号处理器（ISP），称为小型和可学习的ISP模块（SLIM）。SLIM的关键是识别基于物理的isp的瓶颈，并用定制的基于学习的模块取代它们。具体来说，SLIM包括五个创新：(i)基于学习的频率解调，（ii）引导去噪，（iii）学习特征提取，（iv）学习索引，(v)学习滤波。在第一阶段，该团队建议优化SLIM并在现场可编程门阵列（FPGA）上实现它。这包括缩小过滤器的大小和简化索引方案以进一步加快SLIM，引入新的编码器以改进泛化，并通过改进编程和实际数据评估和演示来优化内存、通信和处理。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The broader impact of this Small Business Innovation (SBIR) Phase I project will result from the ability to operate digital image sensors at lower light levels than is currently possible. The technology is expected to be deployable in any mid- to low-level camera device, with potential applications across all industries that leverage camera technology. Consumer applications that would benefit from improved low-light imaging include dashboard cameras and notebook cameras for videoconferencing; military and national security applications include night vision and autonomous navigation; while the technology will also enable improved diagnostic capabilities in medical procedures such as endoscopies. The technology is expected to have a direct impact on workforce development, and deployment of the solution will drive economics in consumer electronics.This Small Business Innovation Research (SBIR) Phase I project aims to achieve photon-limited image denoising using a lightweight algorithm that has the potential to be implemented on a camera chip. Accomplishing this goal requires several technological breakthroughs, collectively leading to a new image signal processor (ISP) known as a Small and Learnable ISP Module (SLIM). The key to SLIM is to identify the bottlenecks of physics-based ISPs and replace them with customized learning-based modules. Specifically, SLIM consists of five innovations: (i) learning-based frequency demodulation, (ii) guided denoising, (iii) learned feature extraction, (iv) learned indexing, and (v) learned filtering. In Phase 1, the team proposes to optimize SLIM and implement it on a field programmable gate array (FPGA). This includes shrinking the size of the filters and streamlining the indexing scheme to further speed up SLIM, introducing new encoders to improve generalization, and optimizing the memory, communication, and processing through improved programming and real-data evaluation and demonstration.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.