SHF: Medium: Cross-Stack Algorithm-Hardware-Systems Optimization Towards Ubiquitous On-Device 3D Intelligence

深度神经网络（DNN）推动的三维（3D）重建有望成为下一个技术颠覆者，并彻底改变人类生活和生产方式的许多方面。预计到2028年将达到15756.4亿美元。这是因为3D重建数据具有丰富的尺度和几何信息，从而使3D智能系统能够实现更好的机器环境理解并执行新的可能功能。随着对3D重建需求的不断增长，日常生活设备的数量一直在蓬勃发展，预计到2030年将达到5000亿。因此，对于将支持3D重建的智能功能引入众多（异构）设备（从无人机到自动驾驶汽车、增强现实（AR）和虚拟现实（VR）设备等）中以实现无处不在的3D智能，存在巨大的需求。尽管无处不在的设备上3D智能有很大的前景，但强大的3D重建算法的复杂性与常用设备中的有限资源之间仍然存在巨大且不断增加的差距。这是因为与2D智能功能相比，3D重建使能的智能功能/模型中所需的增加的数据、模型和训练级复杂性指数地增加了它们的计算成本。此外，与基于2D图像的DNN模型相比，3D重建模型具有非常独特的计算和数据访问模式。该项目旨在缩小上述差距，并通过全面探索算法、硬件和系统级创新的联合优化和协调，促进实现设备上3D重建智能功能的系统性突破。该项目的成果将最终创造创新的课程材料和开放的教育资源，旨在吸引多样化的学生，促进包容性的学习环境，并作为创造力和创新的跳板。该项目将推进知识，并产生科学原理和工具，以实现设备上的3D智能。首先，Thrusts 1和Thrusts 2将开发专用算法硬件协同设计解决方案的基本基础，以实现设备上3D重建的即时训练和实时推理。特别地，考虑到目标场景的足够数量的捕获视图并不总是可用，推力1关注于每场景优化场景，而推力2关注于跨场景。第二，Thrusts 3和4将开发算法-系统协同优化解决方案，以实现资源自适应的设备上3D智能。具体来说，Thrust 3通过扩展到异构边缘设备上跨场景3D重建模型的分布式学习来处理隐私敏感/专有的设备上训练数据，而Thrust 4通过利用算法系统协同设计来探索资源自适应推理调度方法。最后，一个集成的努力进行评估的四个推力的创新，并证明其在现实系统中的好处。总体而言，该项目旨在通过提高模型精度、效率和实现实时推理和学习，显著推进设备上3D重建的最新技术水平。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep neural network (DNN)-boosted three dimensional (3D) reconstruction promises to become the next tech disruptor and revolutionize many aspects of human life and means of production. It is expected to reach a value of $1575.64 billion by 2028. This is because 3D-reconstructed data is rich in scale and geometric information, thus enabling 3D intelligent systems to achieve much better machine-environment understanding and perform newly possible functionalities. In parallel with the growing need for 3D reconstruction, the number of daily-life devices has been booming and is estimated to reach 500 billion by 2030. Therefore, there has been a monumental demand for bringing 3D reconstruction-enabled intelligent functionalities into numerous (heterogeneous) devices, ranging from drones to self-driving cars, augmented reality (AR) and virtual reality (VR) devices, and many more, for enabling ubiquitous 3D intelligence. Despite the big promise of ubiquitous on-device 3D intelligence, a vast and increasing gap still exists between the prohibitive complexity of powerful 3D reconstruction algorithms and the constrained resources in commonly used devices. This is because the increased data-, model-, and training-level complexity required in 3D reconstruction-enabled intelligent functionalities/models exponentially aggravates their computational cost as compared to 2D intelligent ones. Furthermore, 3D reconstruction models feature very unique computational and data access patterns compared to those of 2D image-based DNN models. This project aims to close the aforementioned gap and foster a systematic breakthrough for enabling on-device 3D reconstruction-enabled intelligent functionalities through a holistic exploration of a joint optimization and harmonization of algorithm-, hardware-, and system-level innovations. The results of this project will culminate in the creation of innovative course materials and open educational resources designed to engage a diverse student body, fostering an inclusive learning environment and serving as a springboard for creativity and innovation. The project will advance knowledge and produce scientific principles and tools for enabling on-device 3D intelligence. First, Thrusts 1 and 2 will develop the fundamental underpinnings of dedicated algorithm-hardware co-design solutions for enabling instant training and real-time inference of on-device 3D reconstruction. In particular, Thrust 1 focuses on per-scene optimization scenarios and Thrust 2 concentrates on cross-scene scenarios considering that a sufficient number of captured views of the target scene is not always available. Second, Thrusts 3 and 4 will develop algorithm-system co-optimization solutions for enabling resource-adaptive on-device 3D intelligence. Specifically, Thrust 3 deals with privacy-sensitive/proprietary on-device training data by extending to distributed learning of cross-scene 3D reconstruction models over heterogeneous edge devices, while Thrust 4 explores resource adaptive inference scheduling methods by leveraging algorithm-system co-design. Finally, an integration effort is conducted to evaluate the innovations of the four thrusts and demonstrate their benefits in realistic systems. Overall, this project is geared towards significantly advancing the state-of-the-art of on-device 3D reconstruction by improving model accuracy, efficiency and enabling real-time inference and learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.