Integrating Federated Split Neural Network with Artificial Stereoscopic Compound Eyes for Optical Flow Sensing in 3D Space with Precision

该项目旨在开发一种受节肢动物眼睛启发的创新图像传感器，具有视场广、运行速度快、目标跟踪效率高的特点。这些智能传感器可以极大地扩大视野，提高响应速度，并以能源效率运行。这一进步的核心是光电二极管与人造突触的融合和几何形状的模仿，这种配对反映了昆虫眼睛内神经网络的生物过程。这种配置不仅促进了视觉信息的快速处理，还降低了这样做所需的能量，标志着从传统的平面成像系统向前迈出了重要的一步。与传统系统不同，传感器阵列将采用半球设计，灵感来自大自然自己的广角和高效视觉解决方案。这种几何优化对于以更高的精度和深度捕捉视觉场景中的运动模式至关重要，这使得该技术对于需要精确的运动检测和空间感知的应用来说是非常宝贵的。为了处理这些传感器捕获的高维数据，该项目进一步引入了专门的神经网络结构。这种方法在传感器和中央处理单元之间分配数据处理任务，确保快速、高效和精确的分析。这种配置特别适合需要快速决策的动态环境。通过模拟自然界中发现的复杂视觉系统，它提供了一种对技术和生物学融合的未来的一瞥，提供了有效的解决方案。这项工作的潜在应用跨越了不同的领域，有望提高系统在机器人、自动驾驶等领域的效率和能力，从而推动科学技术的进步，造福社会。该项目的目标是开发和集成一种能够先进的原位目标跟踪和深度感知的新型人工复眼系统。该系统利用了光电二极管与人造突触设备集成的独特优势，体现了用于即时数据处理的计算层，类似于生物突触功能。硬件设计灵感来自节肢动物眼睛的半球结构，实现了大视野和快速图像采集。为了应对高效处理高维视觉数据的挑战，将采用一种通信、存储和节能的联合分裂学习框架。该框架通过在传感器和中央处理单元之间分配计算任务来优化数据处理，显著增强了系统的实时目标跟踪能力。通过将先进的硬件与软件算法相结合，该项目旨在创建一种系统，不仅能促进对生物启发成像的科学理解，还能为现实世界的应用提供实用的解决方案。该项目的成功实施有望建立光学传感技术的新标准，为计算机视觉和神经形态计算领域做出重大贡献。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project is aiming to develop an innovative image sensor inspired by arthropod eyes, featuring a wide field of view, high-speed operation, and efficient object tracking. These smart sensors can dramatically expand the field of view, enhance response speeds, and operate with energy efficiency. Central to this advancement is the fusion of photodiodes with artificial synapses and mimicry of geometrical shape, a pairing that mirrors the biological processes of neural networks within the insect eye. This configuration not only facilitates the rapid processing of visual information but also reduces the energy required to do so, marking a significant step forward from traditional planar imaging systems. Unlike conventional systems, the sensor arrays will adopt hemispherical designs, inspired by nature's own solution to wide-angle and efficient vision. This geometric optimization is crucial for capturing the patterns of movement across a visual scene with enhanced accuracy and depth, making the technology invaluable for applications that require precise motion detection and spatial awareness. To process the high-dimensional data captured by these sensors, the project further introduces a specialized neural network architecture. This approach divides data processing tasks between the sensors and a central processing unit, ensuring swift, efficient, and precise analysis. Such a configuration is particularly suited for dynamic environments where rapid decision-making is essential. By emulating the intricate vision systems found in nature, it offers a glimpse into a future where technology and biology converge, providing impactful solutions. The potential applications of this work span across various sectors, promising to enhance the efficiency, and capabilities of systems in robotics, autonomous driving, and beyond, thereby advancing the progress of science and technology for the benefit of society.The goal of this project is to develop and integrate a novel artificial compound eye system capable of advanced in-situ object tracking and depth perception. This system leverages the unique advantages of a photodiode integrated with an artificial synaptic device, embodying a computational layer for immediate data processing akin to biological synaptic functions. The hardware design, inspired by the hemispherical structure of arthropod eyes, enables a wide field of view and rapid image acquisition. To address the challenges of processing high-dimensional visual data efficiently, a communication, storage, and energy-efficient federated split learning framework will be employed. This framework optimizes data processing by distributing computational tasks between the sensor and a centralized processing unit, significantly enhancing the system's real-time object tracking capabilities. By integrating advanced hardware with software algorithms, this project aims to create a system that not only advances the scientific understanding of bio-inspired imaging but also offers practical solutions for real-world applications. The successful execution of this project is expected to set a new standard in optical sensing technology, contributing significantly to the fields of computer vision and neuromorphic computing.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.