CNS Core: Small: A Split Software Architecture for Enabling High-Quality Mixed Reality on Commodity Mobile Devices

通过将物理和数字世界融合到编程体验中，混合现实（MR）允许用户可视化并与数字信息（如3D叠加和实时数据）进行交互，并在许多社会领域（包括教育，远程工作，军事训练和医疗保健，如远程医疗）中具有重要应用。尽管MR技术具有巨大的潜力，但当今市场上可用的MR解决方案要么是昂贵的企业级，要么是只能支持低质量MR内容的消费级，这导致了较差的用户体验。 当前企业级和消费级MR解决方案的高成本和/或低质量导致MR行业面临根本性的“内容采用”困境：MR内容的缺乏限制了定制MR耳机的市场渗透，MR耳机的低市场渗透反过来又阻碍了MR内容的发展。该NSF CSR项目提案将开发关键技术，以在智能手机等商用移动的设备上实现高质量MR。通过简单的透视头戴式设备（HMD）观看，该设备具有用于输入的高分辨率相机和用于输出的投影仪，例如Nreal Light眼镜。这些技术将把数以百万计的智能手机（配备有上述廉价的HMD）转变为无处不在的MR设备，并且这样做有助于MR行业克服“内容采用”困境，并为MR技术及其许多重要应用的广泛采用铺平道路。该项目旨在创建第一个分离软件架构，使高质量的MR应用程序能够在商品移动的设备上运行;能够联合优化卸载多个基于深度神经网络（DNN）的任务，这些任务构成了一个复杂的资源密集型应用程序，例如在带宽有限且随时间变化的无线网络上的MR;联合调度资源密集型应用（例如MR）的多个基于DNN的任务以有效共享所有本地资源（例如CPU、GPU和新兴移动的设备上的其他处理器（例如NPU））的能力;以及通过在多个移动的设备上扩展拆分软件架构以有效地共享有限的全球资源（如无线网络和边缘云）来支持商品移动的设备上的高质量多人MR的能力。从技术上讲，这项工作预计通过开发通用边缘辅助软件架构，在当前和未来的移动的计算平台（如智能眼镜）上实现延迟敏感的5G/6 G应用程序，从而在商品智能手机上支持AR/VR/MR领域之外产生深远的影响。为MR开发所提出的技术有可能从根本上克服行业面临的“部署-内容”困境，并促进MR技术及其许多社会应用的扩散和广泛采用。通过使智能手机成为发达国家和发展中国家的人们获取信息和AR/VR/MR等新技术的重要推动力，从而成为克服“数字鸿沟”的重要工具，这项工作的重要性将进一步提高。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

By blending the physical and digital worlds into a programmed experience, Mixed Reality (MR) allows users to visualize and interact with digital information such as 3D overlays and real-time data and has important applications in many societal domains including education, remote working, military training, and health care such as tele-medicine. Despite the tremendous potential of the MR technology, the MR solutions available in today’s market are either enterprise-grade which are costly or consumer-grade which can only support low-quality MR content which leads to poor user experience. The high cost and/or low-quality of current enterprise-grade and consumer-grade MR solutions lead to a fundamental “content-adoption” dilemma faced by the MR industry: the lack of MR content has limited the market penetration of custom-made MR headsets, and the low market penetration of MR headsets in turn has hindered the development of MR content. This NSF CSR project proposal will develop key technologies to enable high-quality MR on commodity mobile devices like smartphones, etc.., viewed by a simple see-through head-mount devices (HMD) with a high-resolution camera for input and a projector for output such as Nreal Light glasses. Such technologies will transform millions of smartphones (equipped with the above inexpensive HMDs) into ubiquitous MR devices and in doing so help the MR industry to overcome the “content-adoption” dilemma and pave the way for wide adoption of the MR technology and its many important applications. This project aims to create the first split software architecture that enables high-quality MR applications to run on commodity mobile devices; the capability to jointly optimize offloading multiple Deep Neural Network (DNN)-based tasks constituting a complex, resource-intensive application such as MR over the bandwidth-limited and time-varying wireless network; the capability to jointly schedule multiple DNN-based tasks of resource-intensive applications such as MR to efficiently share all local resources such as the CPU, GPU, and other processors such as NPU on emerging mobile devices; and the capability to support high-quality multi-player MR on commodity mobile devices by scaling the split software architecture across multiple mobile devices to efficiently share the limited global resources such as the wireless network and the edge cloud.The proposed research will have lasting impact on knowledge discovery, the computer industry, and the society. Technically, this work anticipates having far-reaching impacts outside the area of supporting AR/VR/MR on commodity smartphones by developing general edge-assisted software architectures for enabling the class of latency-sensitive 5G/6G applications on current and future mobile computing platforms such as smart glasses. Developing the proposed technologies for MR have the potential to fundamentally overcome the “deployment-content” dilemma faced by the industry as well as fostering the proliferation and wide adoption of MR technologies and its many societal applications. The importance of this work will be further heightened by making smartphones an important enabler of accessing information and new technologies like AR/VR/MR for people in both developed and developing countries and hence being an important tool in overcoming the “digital divide".This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.