RINGS: l-RIM: Learning based Resilient Immersive Media-Compression, Delivery, and Interaction

RINGS：l-RIM：基于学习的弹性沉浸式媒体压缩、交付和交互

• 批准号: 2148382
• 负责人: Shiwen Mao
• 金额: $ 99.33万
• 依托单位: Auburn University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-05-01 至 2025-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2148382&HistoricalAwards=false
• 关键词: RINGS; RIM; Learning; based; Resilient

Augmented Reality/Virtual Reality (AR/VR) has been recognized as a transformative service for the Next Generation (NextG) network systems, while wireless supported AR/VR will offer great flexibility and enhanced immersive experience to users and unleash a plethora of new applications. The primary goal of this project is to explore innovative technologies in NextG and Artificial intelligence (AI) to provide a unified media compression, communication, and computing framework to enable resilient wireless AR/VR. The research agenda comprises five thrusts. Thrusts 1 and 2 are focused on learning-based immersive media compression, i.e., developing high-efficiency Light Field and Point Cloud compression solutions, respectively. The next two thrusts explore the fundamental concepts and techniques that facilitate wireless AR/VR transmission and interaction. Thrust 3 shall develop theoretical models and explore the fundamental performance tradeoffs among compression communication, computation, and caching for wireless AR/VR considering several NextG communications and networking technologies. The work in Thrust 4 shall investigate the application of edge intelligence to develop learning-based algorithms for resilient AR/VR transmissions. The design of resilient AR/VR will be guided by the model-based performance analysis developed in Thrust 3. Thrust 5 will integrate the techniques developed in Thrusts 1-4, and validate their performance with simulation studies using open-source datasets and experimental studies using an AR/VR testbed and publicly available wireless and cloud-related platforms.The proposed research has a high potential to achieve great impacts in our research community and society. The wireless community has been making great efforts on developing learning-based solutions for the NextG wireless and mobile communication, networking, sensing, and computing systems. Findings from this project are expected to facilitate breakthroughs in enabling immersive media applications that require huge amounts of data to train complex deep learning models. The PIs are committed to integration of research and education. This project will offer research and educational opportunities for undergraduate and graduate students at the two collaborating institutions. New training certificate courses will be developed and research outcomes will be incorporated into a forthcoming textbook by PI Mao. The several existing outreach programs at the two institutions will be leveraged to broaden participation from underrepresented groups. Research outcomes will be disseminated through scholarly publications, a project website, open-source repositories, as well as workshops and journal special issues to be organized by the PIs. Industry will also be pursued through the NSF IUCRC Center for Big Learning at The University of Missouri, Kansas City.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

增强现实/虚拟现实（AR/VR）已被公认为下一代（NextG）网络系统的变革性服务，而无线支持的AR/VR将为用户提供极大的灵活性和增强的沉浸式体验，并释放大量新的应用程序。该项目的主要目标是探索NextG和人工智能（AI）中的创新技术，以提供统一的媒体压缩、通信和计算框架，以实现弹性无线AR/VR。研究议程包括五个重点。重点1和重点2是基于学习的沉浸式媒体压缩，即分别开发高效的光场和点云压缩解决方案。接下来的两个重点将探讨促进无线AR/VR传输和交互的基本概念和技术。Thrust 3将开发理论模型，并考虑几种NextG通信和网络技术，探索无线AR/VR压缩通信、计算和缓存之间的基本性能权衡。Thrust 4的工作将研究边缘智能的应用，为弹性AR/VR传输开发基于学习的算法。弹性AR/VR的设计将以Thrust 3开发的基于模型的性能分析为指导。推力5将集成推力1-4中开发的技术，并通过使用开源数据集的仿真研究和使用AR/VR测试平台以及公开可用的无线和云相关平台的实验研究来验证其性能。建议的研究在我们的研究界和社会中具有很大的影响潜力。无线界一直致力于为下一代无线和移动通信、网络、传感和计算系统开发基于学习的解决方案。该项目的研究成果有望促进沉浸式媒体应用的突破，这些应用需要大量数据来训练复杂的深度学习模型。学院致力于将研究与教育相结合。该项目将为两所合作院校的本科生和研究生提供研究和教育机会。将开发新的培训证书课程，并将研究成果纳入皮茂即将出版的教科书中。这两个机构现有的几个外展项目将被用来扩大代表性不足群体的参与。研究成果将通过学术出版物、项目网站、开放源代码存储库以及由pi组织的研讨会和期刊特刊来传播。工业也将通过位于堪萨斯城密苏里大学的NSF - IUCRC大学习中心进行研究。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Multi-stage Locally and Long-range Correlated Feature Fusion for Learned In-loop Filter in VVC

• DOI: 10.1109/vcip56404.2022.10008834
• 发表时间: 2022-12
• 期刊: 2022 IEEE International Conference on Visual Communications and Image Processing (VCIP)
• 作者: B. Kathariya;Zhu Li;Hongtao Wang;G. V. D. Auwera

Location-Dependent Augmented Reality Services in Wireless Edge-Enabled Metaverse Systems

• DOI: 10.1109/ojcoms.2023.3234254
• 发表时间: 2023
• 期刊: IEEE Open Journal of the Communications Society
• 影响因子: 7.9
• 作者: Hongliang Zhang;S. Mao;D. Niyato;Zhu Han

LoneNeuron: A Highly-Effective Feature-Domain Neural Trojan Using Invisible and Polymorphic Watermarks

• DOI: 10.1145/3548606.3560678
• 发表时间: 2022-11
• 期刊: Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security
• 作者: Zeyan Liu;Fengjun Li;Zhu Li;Bo Luo

Enabling Mobile Virtual Reality with Open 5G, Fog Computing and Reinforcement Learning

• DOI: 10.1109/mnet.010.2100481
• 发表时间: 2022-11
• 期刊: IEEE Network
• 影响因子: 9.3
• 作者: Yaohua Sun;Jianmin Chen;Zeyu Wang;M. Peng;S. Mao

Multi-Stage Spatial and Frequency Feature Fusion using Transformer in CNN-Based In-Loop Filter for VVC

• DOI: 10.1109/pcs56426.2022.10017998
• 发表时间: 2022-12
• 期刊: 2022 Picture Coding Symposium (PCS)
• 作者: B. Kathariya;Zhu Li;Hongtao Wang;Mohammad Coban