Collaborative Research: SHF: Small: Software Hardware Architecture Co-Design for Enabling True Virtual Reality on Mobile Devices

虚拟现实(VR)为用户提供了身临其境的物理感受。Metverse等新兴的VR应用程序旨在实现真实人类世界的虚拟对应物，这与物理世界没有什么区别，可以完全模拟真实的人类生活。提供如此真实的VR体验带来了前所未有的计算需求，并给资源受限的移动VR设备带来了巨大的挑战。因为仅通过重新设计移动软件或硬件系统不能满足计算需求。该项目通过软硬件架构协同设计解决了这些挑战。由于VR被认为是一种新兴技术，可以从根本上改变人们与环境和彼此互动的方式，该项目通过促进VR系统的大规模采用进入人们日常生活的方方面面，具有造福社会的巨大潜力。该项目还通过开发新课程、传播教育和培训研究、吸引未被充分代表的学生参与研究以及向高中生提供外展服务来为社会做出贡献。该项目的主要目标是通过充分了解和利用不同VR软件系统和硬件组件之间的密切关联，在移动设备上提供真实的VR体验。该项目包括三个研究任务：(1)开发新的VR计算技术和GPU硬件设计，通过自适应地识别不同用户绘制的VR帧之间的冗余并尽可能地重用这些帧，来最小化VR计算量；(2)根据不同VR用户在视场中的共性，研究VR帧平铺技术，根据移动硬件的特点，最大限度地减少移动设备接收和处理VR帧的能耗；(3)探索不同种类的移动硬件，特别是人工智能加速器的潜力，以在移动设备上实现基于人工智能的图像升级，以补偿通信链路变化可能导致的VR质量下降。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Virtual Reality (VR) provides users with immersive senses of physical presence in the virtual world. Emerging VR applications such as Metaverse aim to realize a virtual counterpart of the real human world, which is truly indistinguishable from the physical world and allows complete simulation of real human life. Providing such true VR experiences results in an unprecedented amount of computation demand, and creates tremendous challenges on resource-constrained mobile VR devices. Since the computation demand cannot be met by only redesigning either the mobile software or hardware systems. This project addresses these challenges via software-hardware architecture co-design. As VR is considered as one emerging technique that can fundamentally transform the way people interact with the environment and each other, this project has great potential to benefit society by facilitating large-scale adoption of VR systems into every aspect of people’s daily lives. This project is also contributing to society through developing new curricula, disseminating research for education and training, engaging under-represented students in research, and outreaching to high-school students.The primary goal of this project is to provide true VR experiences on mobile devices by fully understanding and utilizing the close correlation between different VR software systems and hardware components. The project consists of three research tasks: (1) developing new VR computing techniques and GPU hardware designs to minimize the amount of VR computation, by adaptively identifying the redundancy among the rendered VR frames of different users and reusing these frames whenever possible; (2) investigating VR frame tiling techniques based on different VR users’ commonality in Field of Views, to minimize the mobile device’s energy consumption on receiving and processing VR frames according to the mobile hardware characteristics; (3) exploring the potential of heterogeneous mobile hardware, especially AI accelerators, to enable AI-based image upscaling on mobile devices that compensates the possible VR quality degradation due to communication link variations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.