Collaborative Research: SHF: Small: Software Hardware Architecture Co-Design for Enabling True Virtual Reality on Mobile Devices

点击翻译按钮获取中文摘要

Virtual Reality (VR) provides users with immersive senses of physical presence in the virtual world. Emerging VR applications such as Metaverse aim to realize a virtual counterpart of the real human world, which is truly indistinguishable from the physical world and allows complete simulation of real human life. Providing such true VR experiences results in an unprecedented amount of computation demand, and creates tremendous challenges on resource-constrained mobile VR devices. Since the computation demand cannot be met by only redesigning either the mobile software or hardware systems. This project addresses these challenges via software-hardware architecture co-design. As VR is considered as one emerging technique that can fundamentally transform the way people interact with the environment and each other, this project has great potential to benefit society by facilitating large-scale adoption of VR systems into every aspect of people’s daily lives. This project is also contributing to society through developing new curricula, disseminating research for education and training, engaging under-represented students in research, and outreaching to high-school students.The primary goal of this project is to provide true VR experiences on mobile devices by fully understanding and utilizing the close correlation between different VR software systems and hardware components. The project consists of three research tasks: (1) developing new VR computing techniques and GPU hardware designs to minimize the amount of VR computation, by adaptively identifying the redundancy among the rendered VR frames of different users and reusing these frames whenever possible; (2) investigating VR frame tiling techniques based on different VR users’ commonality in Field of Views, to minimize the mobile device’s energy consumption on receiving and processing VR frames according to the mobile hardware characteristics; (3) exploring the potential of heterogeneous mobile hardware, especially AI accelerators, to enable AI-based image upscaling on mobile devices that compensates the possible VR quality degradation due to communication link variations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.