CNS Core: Small: A Split Software Architecture for Enabling High-Quality Mixed Reality on Commodity Mobile Devices

CNS 核心：小型：用于在商用移动设备上实现高质量混合现实的分离式软件架构

• 批准号: 2112778
• 负责人: Charlie Hu
• 金额: $ 42.44万
• 依托单位: Purdue University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2112778&HistoricalAwards=false
• 关键词: CNS; Core; Small; Split; Software

By blending the physical and digital worlds into a programmed experience, Mixed Reality (MR) allows users to visualize and interact with digital information such as 3D overlays and real-time data and has important applications in many societal domains including education, remote working, military training, and health care such as tele-medicine. Despite the tremendous potential of the MR technology, the MR solutions available in today’s market are either enterprise-grade which are costly or consumer-grade which can only support low-quality MR content which leads to poor user experience. The high cost and/or low-quality of current enterprise-grade and consumer-grade MR solutions lead to a fundamental “content-adoption” dilemma faced by the MR industry: the lack of MR content has limited the market penetration of custom-made MR headsets, and the low market penetration of MR headsets in turn has hindered the development of MR content. This NSF CSR project proposal will develop key technologies to enable high-quality MR on commodity mobile devices like smartphones, etc.., viewed by a simple see-through head-mount devices (HMD) with a high-resolution camera for input and a projector for output such as Nreal Light glasses. Such technologies will transform millions of smartphones (equipped with the above inexpensive HMDs) into ubiquitous MR devices and in doing so help the MR industry to overcome the “content-adoption” dilemma and pave the way for wide adoption of the MR technology and its many important applications. This project aims to create the first split software architecture that enables high-quality MR applications to run on commodity mobile devices; the capability to jointly optimize offloading multiple Deep Neural Network (DNN)-based tasks constituting a complex, resource-intensive application such as MR over the bandwidth-limited and time-varying wireless network; the capability to jointly schedule multiple DNN-based tasks of resource-intensive applications such as MR to efficiently share all local resources such as the CPU, GPU, and other processors such as NPU on emerging mobile devices; and the capability to support high-quality multi-player MR on commodity mobile devices by scaling the split software architecture across multiple mobile devices to efficiently share the limited global resources such as the wireless network and the edge cloud.The proposed research will have lasting impact on knowledge discovery, the computer industry, and the society. Technically, this work anticipates having far-reaching impacts outside the area of supporting AR/VR/MR on commodity smartphones by developing general edge-assisted software architectures for enabling the class of latency-sensitive 5G/6G applications on current and future mobile computing platforms such as smart glasses. Developing the proposed technologies for MR have the potential to fundamentally overcome the “deployment-content” dilemma faced by the industry as well as fostering the proliferation and wide adoption of MR technologies and its many societal applications. The importance of this work will be further heightened by making smartphones an important enabler of accessing information and new technologies like AR/VR/MR for people in both developed and developing countries and hence being an important tool in overcoming the “digital divide".This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

通过将物理和数字世界融合到编程体验中，混合现实(MR)允许用户可视化数字信息并与其交互，如3D覆盖和实时数据，并在许多社会领域具有重要应用，包括教育、远程工作、军事训练和远程医疗等医疗保健。尽管MR技术具有巨大的潜力，但当今市场上可用的MR解决方案要么是成本高昂的企业级解决方案，要么是只能支持低质量MR内容的消费级解决方案，这会导致糟糕的用户体验。当前企业级和消费级MR解决方案的高成本和/或低质量导致MR行业面临一个根本性的“内容采用”困境：MR内容的缺乏限制了定制MR耳机的市场渗透率，而MR耳机的低市场渗透率反过来又阻碍了MR内容的发展。这项NSF CSR项目提案将开发关键技术，使智能手机等商用移动设备能够实现高质量的MR，通过简单的透明头戴式设备(HMD)查看，HMD具有用于输入的高分辨率摄像头和用于输出的投影仪，如NrealLight眼镜。这些技术将把数以百万计的智能手机(配备上述廉价的HMD)转变为无处不在的MR设备，从而帮助MR行业克服“内容采用”困境，并为MR技术及其许多重要应用的广泛采用铺平道路。该项目旨在创建第一个拆分式软件体系结构，使高质量的MR应用程序能够在商用移动设备上运行；能够联合优化卸载构成带宽有限和时变无线网络上的MR等复杂的资源密集型应用程序的多个基于深度神经网络(DNN)的任务；能够联合调度MR等资源密集型应用程序的多个基于DNN的任务，以高效共享新兴移动设备上的所有本地资源，如CPU、GPU和其他处理器(如NPU)；以及通过跨多个移动设备扩展拆分的软件架构以高效共享有限的全球资源(如无线网络和边缘云)来支持商用移动设备上的高质量多玩家MR的能力。从技术上讲，这项工作预计将通过开发通用边缘辅助软件架构，在当前和未来的移动计算平台(如智能眼镜)上支持延迟敏感型5G/6G应用，从而在支持商用智能手机上的AR/VR/MR领域之外产生深远的影响。为MR开发拟议的技术有可能从根本上克服行业面临的“部署-内容”困境，并促进MR技术及其许多社会应用的扩散和广泛采用。这项工作的重要性将进一步提高，使智能手机成为发达国家和发展中国家人民获取信息和AR/VR/MR等新技术的重要手段，从而成为克服“数字鸿沟”的重要工具。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Poster: BystandAR: Protecting Bystander Visual Data in Augmented Reality Systems

• DOI: 10.1145/3581791.3597377
• 发表时间: 2023-06
• 期刊: Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services
• 作者: Matthew Corbett;Brendan David-John;Jiacheng Shang;Y. C. Hu;Bo Ji

BystandAR: Protecting Bystander Visual Data in Augmented Reality Systems

• DOI: 10.1145/3581791.3596830
• 发表时间: 2023-06
• 期刊: Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services
• 作者: Brendan David-John;Jiacheng Shang;Bo Ji;Matthew Corbett;Y. C. Hu;Bo Ji. 2023. BystandAR

Do Larger (More Accurate) Deep Neural Network Models Help in Edge-assisted Augmented Reality?

更大（更准确）的深度神经网络模型有助于边缘辅助增强现实吗？

• DOI: 10.1145/3472727.3472807
• 发表时间: 2021
• 期刊: NAI'21: Proceedings of the ACM SIGCOMM 2021 Workshop on Network-Application Integration
• 作者: Meng, Jiayi;Kong, Zhaoning;Xu, Qiang;Hu, Y. Charlie
• 通讯作者: Hu, Y. Charlie

An In-Depth Study of Uplink Performance of 5G mmWave Networks

5G毫米波网络上行链路性能的深入研究

• DOI: 10.1145/3538394.3546042
• 发表时间: 2022
• 期刊: and Use Cases (5G-MeMU
• 作者: Moinak Ghoshal, Z. Jonny

Can 5G mmWave Support Multi-user AR?

5G毫米波能否支持多用户AR？

• DOI: 10.1007/978-3-030-98785-5_8
• 发表时间: 2022
• 期刊: vol 13210
• 作者: Moinak Ghoshal, Pranab Dash