SHF: Medium: Cross-Stack Algorithm-Hardware-Systems Optimization Towards Ubiquitous On-Device 3D Intelligence

SHF：中：跨堆栈算法-硬件-系统优化，实现无处不在的设备上 3D 智能

• 批准号: 2312758
• 负责人: Yingyan Lin
• 金额: $ 119.84万
• 依托单位: Georgia Tech Research Corporation
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-07-01 至 2027-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2312758&HistoricalAwards=false
• 关键词: SHF; Medium; Cross; Stack; Algorithm

Deep neural network (DNN)-boosted three dimensional (3D) reconstruction promises to become the next tech disruptor and revolutionize many aspects of human life and means of production. It is expected to reach a value of $1575.64 billion by 2028. This is because 3D-reconstructed data is rich in scale and geometric information, thus enabling 3D intelligent systems to achieve much better machine-environment understanding and perform newly possible functionalities. In parallel with the growing need for 3D reconstruction, the number of daily-life devices has been booming and is estimated to reach 500 billion by 2030. Therefore, there has been a monumental demand for bringing 3D reconstruction-enabled intelligent functionalities into numerous (heterogeneous) devices, ranging from drones to self-driving cars, augmented reality (AR) and virtual reality (VR) devices, and many more, for enabling ubiquitous 3D intelligence. Despite the big promise of ubiquitous on-device 3D intelligence, a vast and increasing gap still exists between the prohibitive complexity of powerful 3D reconstruction algorithms and the constrained resources in commonly used devices. This is because the increased data-, model-, and training-level complexity required in 3D reconstruction-enabled intelligent functionalities/models exponentially aggravates their computational cost as compared to 2D intelligent ones. Furthermore, 3D reconstruction models feature very unique computational and data access patterns compared to those of 2D image-based DNN models. This project aims to close the aforementioned gap and foster a systematic breakthrough for enabling on-device 3D reconstruction-enabled intelligent functionalities through a holistic exploration of a joint optimization and harmonization of algorithm-, hardware-, and system-level innovations. The results of this project will culminate in the creation of innovative course materials and open educational resources designed to engage a diverse student body, fostering an inclusive learning environment and serving as a springboard for creativity and innovation. The project will advance knowledge and produce scientific principles and tools for enabling on-device 3D intelligence. First, Thrusts 1 and 2 will develop the fundamental underpinnings of dedicated algorithm-hardware co-design solutions for enabling instant training and real-time inference of on-device 3D reconstruction. In particular, Thrust 1 focuses on per-scene optimization scenarios and Thrust 2 concentrates on cross-scene scenarios considering that a sufficient number of captured views of the target scene is not always available. Second, Thrusts 3 and 4 will develop algorithm-system co-optimization solutions for enabling resource-adaptive on-device 3D intelligence. Specifically, Thrust 3 deals with privacy-sensitive/proprietary on-device training data by extending to distributed learning of cross-scene 3D reconstruction models over heterogeneous edge devices, while Thrust 4 explores resource adaptive inference scheduling methods by leveraging algorithm-system co-design. Finally, an integration effort is conducted to evaluate the innovations of the four thrusts and demonstrate their benefits in realistic systems. Overall, this project is geared towards significantly advancing the state-of-the-art of on-device 3D reconstruction by improving model accuracy, efficiency and enabling real-time inference and learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度神经网络（DNN）增强了三维（3D）重建有望成为下一个技术破坏者，并彻底改变了人类生活和生产手段的许多方面。预计到2028年，它的价值将达到15756.4亿美元。这是因为3D重建数据的规模和几何信息丰富，从而使3D Intelligent Systems能够获得更好的机器环境理解并执行新可能的功能。随着3D重建的需求不断增长，日常生活的数量一直在蓬勃发展，到2030年到2030年估计将达到5000亿次。因此，将3D重建功能的智能功能带入了许多（异质）设备，从无人机到更多（自我驾驶的汽车），以及VR的现实（VR）现实（VR），AR（AR）的现实（AR）竞争（AR），这已经有着巨大的需求。实现无处不在的3D智能。尽管无处不在的机上3D智能有很大的希望，但在强大的3D重建算法的过度复杂性与常用设备中的约束资源之间仍然存在巨大而增加的差距。这是因为与2D Intelligent相比，在3D重建的智能功能/模型中，增加的数据，模型和培训水平复杂性增加了其计算成本。此外，与基于2D图像的DNN模型相比，3D重建模型具有非常独特的计算和数据访问模式。该项目旨在缩小上述差距，并促进系统的突破，以通过对算法，硬件 - 和系统级创新的联合优化和协调的整体探索和协调，从而实现了启用启用的智障3D重建智能功能。该项目的结果将在创造创新的课程材料和开放的教育资源中达到顶峰，旨在参与多元化的学生团体，培养包容性的学习环境并充当创造力和创新的跳板。该项目将促进知识，并生产科学原理和工具，以实现在设备3D智能上。首先，推力1和2将开发专用算法 - 硬件联合设计解决方案的基本基础，以实现即时培训和实时推断3D重建。特别是，考虑到目标场景的足够数量的捕获视图并非总是可用，推力1专注于每场景优化方案和推力2集中在跨场景方案上。其次，推力3和4将开发算法 - 系统合作解决方案，以实现资源适应性的运动3D智能。具体而言，通过将3次与对隐私敏感/专有的机上培训数据进行交易，通过将跨场所3D重建模型的分布式学习扩展到异质边缘设备上，而Throust 4通过利用AlgorithM-System Sysy Sy-Sysignsign来探索资源自适应推理调度方法。最后，进行了一项集成努力，以评估这四个推力的创新，并证明它们在现实系统中的好处。总体而言，该项目旨在通过提高模型的准确性，效率并实现实时推理和学习来显着提高处式3D重建的最先进。这项奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子优点和更广泛影响的评估来通过评估来支持的。