Collaborative Research: SHF: Small: Software Hardware Architecture Co-Design for Enabling True Virtual Reality on Mobile Devices

虚拟现实（VR）为用户提供了虚拟世界中物理存在的沉浸感。新兴的虚拟现实应用，如Metaverse，旨在实现真实的人类世界的虚拟对应物，这是真正无法区分的物理世界，并允许完全模拟真实的人类生活。 提供这种真实的VR体验会产生前所未有的计算需求，并对资源受限的移动的VR设备造成巨大挑战。由于计算需求不能满足仅仅重新设计的移动的软件或硬件系统，该项目通过软件-硬件架构协同设计来解决这些挑战。由于VR被认为是一种新兴技术，可以从根本上改变人们与环境和彼此之间的互动方式，因此该项目具有巨大的潜力，可以通过促进VR系统大规模应用于人们日常生活的各个方面来造福社会。此外，本项目还通过开发新课程、传播教育培训研究成果、让代表性不足的学生参与研究、面向高中生开展外展等方式，为社会做出贡献。本项目的主要目标是充分理解和利用VR软件系统和硬件之间的紧密关联，在移动的设备上提供真正的VR体验。该项目包括三个研究任务：（1）开发新的VR计算技术和GPU硬件设计，通过自适应地识别不同用户渲染的VR帧之间的冗余并尽可能重用这些帧，以最大限度地减少VR计算量;（2）研究了基于不同VR用户视野共性的VR框架拼接技术，根据所述移动的硬件特性，最小化所述移动终端在接收和处理VR帧时的能量消耗;（3）探索异构移动的硬件的潜力，尤其是AI加速器，使AI-基于移动的设备上的图像放大，以补偿由于通信链路变化而可能导致的VR质量下降。该奖项反映了NSF的法定使命，通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Virtual Reality (VR) provides users with immersive senses of physical presence in the virtual world. Emerging VR applications such as Metaverse aim to realize a virtual counterpart of the real human world, which is truly indistinguishable from the physical world and allows complete simulation of real human life. Providing such true VR experiences results in an unprecedented amount of computation demand, and creates tremendous challenges on resource-constrained mobile VR devices. Since the computation demand cannot be met by only redesigning either the mobile software or hardware systems, this project addresses these challenges via software-hardware architecture co-design. As VR is considered as one emerging technique that can fundamentally transform the way people interact with the environment and each other, this project has great potential to benefit society by facilitating large-scale adoption of VR systems into every aspect of people’s daily lives. This project is also contributing to society through developing new curricula, disseminating research for education and training, engaging under-represented students in research, and outreaching to high-school students.The primary goal of this project is to provide true VR experiences on mobile devices by fully understanding and utilizing the close correlation between different VR software systems and hardware components. The project consists of three research tasks: (1) developing new VR computing techniques and GPU hardware designs to minimize the amount of VR computation, by adaptively identifying the redundancy among the rendered VR frames of different users and reusing these frames whenever possible; (2) investigating VR frame tiling techniques based on different VR users’ commonality in Field of Views, to minimize the mobile device’s energy consumption on receiving and processing VR frames according to the mobile hardware characteristics; (3) exploring the potential of heterogeneous mobile hardware, especially AI accelerators, to enable AI-based image upscaling on mobile devices that compensates the possible VR quality degradation due to communication link variations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.