SHF: Medium: Cross-Stack Algorithm-Hardware-Systems Optimization Towards Ubiquitous On-Device 3D Intelligence

深度神经网络(DNN)推动的三维(3D)重建有望成为下一个技术颠覆者，并彻底改变人类生活和生产方式的许多方面。预计到2028年，其价值将达到15756.4亿美元。这是因为3D重建的数据具有丰富的尺度和几何信息，从而使3D智能系统能够更好地理解机器环境并执行新的可能功能。在对3D重建需求日益增长的同时，日常生活设备的数量一直在蓬勃发展，预计到2030年将达到5000亿台。因此，将支持3D重建的智能功能引入从无人机到自动驾驶汽车、增强现实(AR)和虚拟现实(VR)设备等众多(异类)设备，以实现无处不在的3D智能，一直是一个巨大的需求。尽管无处不在的设备上3D智能前景光明，但强大的3D重建算法令人望而却步的复杂性与常用设备中有限的资源之间仍存在巨大且不断扩大的差距。这是因为与2D智能功能/模型相比，启用3D重建的智能功能/模型所需的增加的数据、模型和训练级别的复杂性以指数方式加重了它们的计算成本。此外，与基于2D图像的DNN模型相比，3D重建模型具有非常独特的计算和数据访问模式。该项目旨在通过全面探索算法、硬件和系统级创新的联合优化和协调，缩小上述差距，并在实现设备上3D重建启用智能功能方面实现系统性突破。该项目的最终成果将是创建创新的课程材料和开放的教育资源，旨在吸引不同的学生群体，促进包容性的学习环境，并成为创造力和创新的跳板。该项目将促进知识的发展，并产生科学原理和工具，以实现设备上的3D智能。首先，推进器1和推进器2将开发专用算法-硬件联合设计解决方案的基本基础，以实现设备上3D重建的即时培训和实时推理。特别是，推力1专注于每个场景的优化场景，而推力2专注于跨场景场景，考虑到目标场景的足够数量的捕获视图并不总是可用的。其次，Thrust3和Thrust4将开发算法系统联合优化解决方案，以实现资源自适应的设备上3D智能。具体地说，推力3通过扩展到跨场景3D重建模型在不同边缘设备上的分布式学习来处理隐私敏感/专有设备上的训练数据，而推力4通过利用算法-系统协同设计来探索资源自适应推理调度方法。最后，对四种推力的创新进行了综合评价，并在现实系统中展示了它们的效益。总体而言，该项目旨在通过提高模型的准确性、效率并实现实时推理和学习，显著推进设备上3D重建的最先进水平。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep neural network (DNN)-boosted three dimensional (3D) reconstruction promises to become the next tech disruptor and revolutionize many aspects of human life and means of production. It is expected to reach a value of $1575.64 billion by 2028. This is because 3D-reconstructed data is rich in scale and geometric information, thus enabling 3D intelligent systems to achieve much better machine-environment understanding and perform newly possible functionalities. In parallel with the growing need for 3D reconstruction, the number of daily-life devices has been booming and is estimated to reach 500 billion by 2030. Therefore, there has been a monumental demand for bringing 3D reconstruction-enabled intelligent functionalities into numerous (heterogeneous) devices, ranging from drones to self-driving cars, augmented reality (AR) and virtual reality (VR) devices, and many more, for enabling ubiquitous 3D intelligence. Despite the big promise of ubiquitous on-device 3D intelligence, a vast and increasing gap still exists between the prohibitive complexity of powerful 3D reconstruction algorithms and the constrained resources in commonly used devices. This is because the increased data-, model-, and training-level complexity required in 3D reconstruction-enabled intelligent functionalities/models exponentially aggravates their computational cost as compared to 2D intelligent ones. Furthermore, 3D reconstruction models feature very unique computational and data access patterns compared to those of 2D image-based DNN models. This project aims to close the aforementioned gap and foster a systematic breakthrough for enabling on-device 3D reconstruction-enabled intelligent functionalities through a holistic exploration of a joint optimization and harmonization of algorithm-, hardware-, and system-level innovations. The results of this project will culminate in the creation of innovative course materials and open educational resources designed to engage a diverse student body, fostering an inclusive learning environment and serving as a springboard for creativity and innovation. The project will advance knowledge and produce scientific principles and tools for enabling on-device 3D intelligence. First, Thrusts 1 and 2 will develop the fundamental underpinnings of dedicated algorithm-hardware co-design solutions for enabling instant training and real-time inference of on-device 3D reconstruction. In particular, Thrust 1 focuses on per-scene optimization scenarios and Thrust 2 concentrates on cross-scene scenarios considering that a sufficient number of captured views of the target scene is not always available. Second, Thrusts 3 and 4 will develop algorithm-system co-optimization solutions for enabling resource-adaptive on-device 3D intelligence. Specifically, Thrust 3 deals with privacy-sensitive/proprietary on-device training data by extending to distributed learning of cross-scene 3D reconstruction models over heterogeneous edge devices, while Thrust 4 explores resource adaptive inference scheduling methods by leveraging algorithm-system co-design. Finally, an integration effort is conducted to evaluate the innovations of the four thrusts and demonstrate their benefits in realistic systems. Overall, this project is geared towards significantly advancing the state-of-the-art of on-device 3D reconstruction by improving model accuracy, efficiency and enabling real-time inference and learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.