Collaborative Research: SHF: Small: Software Hardware Architecture Co-Design for Enabling True Virtual Reality on Mobile Devices

虚拟现实（VR）为用户提供身临其境的虚拟世界物理存在感。新兴的虚拟现实应用，如Metaverse，旨在实现真实人类世界的虚拟对应物，与物理世界真正无法区分，可以完全模拟真实的人类生活。提供如此真实的VR体验会导致前所未有的计算需求，并对资源有限的移动VR设备产生巨大挑战。由于计算需求不能仅仅通过重新设计移动软件或硬件系统来满足，因此本项目通过软硬件架构协同设计来解决这些挑战。由于VR被认为是一种新兴技术，可以从根本上改变人们与环境和彼此互动的方式，因此该项目具有巨大的潜力，可以促进VR系统大规模应用于人们日常生活的方方面面，从而造福社会。该项目还通过开发新课程、传播教育和培训方面的研究成果、让代表性不足的学生参与研究以及向高中生伸出援手，为社会做出贡献。该项目的主要目标是通过充分理解和利用不同VR软件系统和硬件组件之间的紧密联系，在移动设备上提供真正的VR体验。该项目包括三个研究任务：(1)开发新的VR计算技术和GPU硬件设计，通过自适应识别不同用户渲染的VR帧之间的冗余并尽可能重用这些帧，从而最大限度地减少VR计算量；(2)研究基于不同VR用户在视场中的共性的VR帧平铺技术，根据移动设备硬件特点，最大限度地降低移动设备接收和处理VR帧的能耗；(3)探索异构移动硬件的潜力，特别是人工智能加速器，使基于人工智能的图像在移动设备上升级，以补偿由于通信链路变化而可能导致的VR质量下降。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Virtual Reality (VR) provides users with immersive senses of physical presence in the virtual world. Emerging VR applications such as Metaverse aim to realize a virtual counterpart of the real human world, which is truly indistinguishable from the physical world and allows complete simulation of real human life. Providing such true VR experiences results in an unprecedented amount of computation demand, and creates tremendous challenges on resource-constrained mobile VR devices. Since the computation demand cannot be met by only redesigning either the mobile software or hardware systems, this project addresses these challenges via software-hardware architecture co-design. As VR is considered as one emerging technique that can fundamentally transform the way people interact with the environment and each other, this project has great potential to benefit society by facilitating large-scale adoption of VR systems into every aspect of people’s daily lives. This project is also contributing to society through developing new curricula, disseminating research for education and training, engaging under-represented students in research, and outreaching to high-school students.The primary goal of this project is to provide true VR experiences on mobile devices by fully understanding and utilizing the close correlation between different VR software systems and hardware components. The project consists of three research tasks: (1) developing new VR computing techniques and GPU hardware designs to minimize the amount of VR computation, by adaptively identifying the redundancy among the rendered VR frames of different users and reusing these frames whenever possible; (2) investigating VR frame tiling techniques based on different VR users’ commonality in Field of Views, to minimize the mobile device’s energy consumption on receiving and processing VR frames according to the mobile hardware characteristics; (3) exploring the potential of heterogeneous mobile hardware, especially AI accelerators, to enable AI-based image upscaling on mobile devices that compensates the possible VR quality degradation due to communication link variations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.