Image sensing with multilayer nonlinear optical neural networks
Image sensing with multilayer nonlinear optical neural networks
复制标题
DOI：
10.1038/s41566-023-01170-8
复制
发表时间：
2023-03-23
期刊：
影响因子：
35
通讯作者：
McMahon, Peter L.
中科院分区：
文献类型：
--
作者：
Wang, Tianyu;Sohoni, Mandar M.;McMahon, Peter L.
基于图像增强器的非线性光学神经网络图像传感器可以对各种机器视觉任务进行高效的全光学图像编码。光学成像通常用于工业和学术界的科学和技术应用。在图像传感中，通过对数字化图像的计算分析来执行诸如物体位置或轮廓的测量。一种新兴的图像传感范例依赖于光学系统，这些光学系统充当编码器，通过提取显著特征将图像光学压缩到低维空间；然而，这些编码器的性能通常受到它们的线性的限制。本文报道了一种用于图像传感的多层非线性光学神经网络(ONN)编码器，该编码器基于商用像增强器作为光学-光学非线性激活函数。这种非线性ONN在几个具有代表性的任务中的表现优于类似尺寸的线性光学编码器，包括机器视觉基准、流式细胞仪图像分类和三维打印真实场景中对象的识别。对于机器视觉任务，特别是那些具有非相干宽带照明的任务，我们的概念允许显著降低对相机分辨率和电子后处理复杂性的要求。一般来说，使用ONN的图像预处理应该能够使图像传感应用以更少的像素、更少的光子、更高的吞吐量和更低的延迟准确地运行。
A nonlinear optical neural network image sensor based on an image intensifier enables efficient all-optical image encoding for a variety of machine-vision tasks.Optical imaging is commonly used for both scientific and technological applications across industry and academia. In image sensing, a measurement, such as of an object's position or contour, is performed by computational analysis of a digitized image. An emerging image-sensing paradigm relies on optical systems that-instead of performing imaging-act as encoders that optically compress images into low-dimensional spaces by extracting salient features; however, the performance of these encoders is typically limited by their linearity. Here we report a nonlinear, multilayer optical neural network (ONN) encoder for image sensing based on a commercial image intensifier as an optical-to-optical nonlinear activation function. This nonlinear ONN outperforms similarly sized linear optical encoders across several representative tasks, including machine-vision benchmarks, flow-cytometry image classification and identification of objects in a three-dimensionally printed real scene. For machine-vision tasks, especially those featuring incoherent broadband illumination, our concept allows for a considerable reduction in the requirement of camera resolution and electronic post-processing complexity. In general, image pre-processing with ONNs should enable image-sensing applications that operate accurately with fewer pixels, fewer photons, higher throughput and lower latency.