Collaborative Research: NeTS: Small: A Privacy-Aware Human-Centered QoE Assessment Framework for Immersive Videos

沉浸式视频，也称为360度视频，为观众提供了对其环境的完整视觉视角。到2025年，其全球市场预计将达到220亿美元。随着360度视频的日益普及，网络运营商和服务提供商越来越热衷于深入了解用户的体验质量（QoE）感知。然而，为2D视频设计的传统QoE模型无法捕捉用户在观看这种新型视频时的独特主观感受。为了弥合这一差距，该项目专注于构建以人为本的模型，通过各种虚拟现实（VR）机载传感器利用“人为因素”。利用多模态感官读数，研究人员将利用用户感知体验的直接表示。这种方法捕获了通常被现有QoE模型中常用的客观系统参数遗漏的主观感觉。这个项目的成功将提供一个深入了解用户的细微差别的感知体验，同时与360度视频。它将为利益相关者创造机会，实施特定于用户的网络资源优化和视频流策略，最终提高客户对个人的满意度。该研究成果将为以人为中心的传感和网络研究社区做出重大贡献，并使360度视频以外的众多VR应用受益。第一个推力研究如何从多模态感官读数中提取显著特征，以进行有效的QoE评估。为了消除标签分布不均匀的数据集所产生的偏差，将开发新的方法，通过探索跨传感模式的有用信息来增加代表性不足的数据。第二个推力寻求优化用于QoE评估的系统资源利用。多个个性化模型将在边缘服务器上共享公共神经网络层，以实现资源高效的模型托管。自适应采样将应用于VR终端在感官数据采集，以保持数据的效用。第三个重点是在不牺牲QoE评估准确性的情况下设计数据隐私保护机制。在差分隐私框架下，该方法的特点是形式化量化的多模态感知数据的数据相关性，这在很大程度上被以前的工作所忽视。拟议的研究是跨学科的，涵盖数据驱动的建模，传感和隐私保护计算。建议的机制和设计将通过测量活动，模拟和实验研究的组合进行全面评估。该奖项反映了NSF的法定使命，并被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

Immersive videos, also known as 360-degree videos, provide viewers with a complete visual perspective of their environment. Its global market is projected to reach 22 billion U.S. dollars by 2025. With the rising popularity of 360-degree videos, network operators and service providers are increasingly keen on gaining insights into users’ Quality of Experience (QoE) perception. However, the conventional QoE models designed for 2D videos fall short of capturing users’ unique subjective feelings in viewing this new type of video. To bridge this gap, the project focuses on constructing human-centered models that utilize “human factors” via a variety of virtual reality (VR) onboard sensors. Leveraging multi-modal sensory readings, the researchers will tap into a direct representation of users’ perceptual experiences. This approach captures subjective feelings often missed by the objective system parameters commonly used in existing QoE models. The success of this project will provide an in-depth understanding of users’ nuanced perceptual experience while engaging with 360-degree videos. It will create opportunities for stakeholders to implement user-specific network resource optimization and video streaming strategies, ultimately enhancing customer satisfaction on an individual basis. The research outcomes will significantly contribute to the human-centered sensing and networking research communities and benefit numerous VR applications beyond 360-degree videos.This project involves three closely related research thrusts. The first thrust investigates how to extract salient features from multi-modal sensory readings for effective QoE assessment. To combat bias arising from datasets with unevenly distributed labels, novel approaches will be developed to augment underrepresented data by exploring useful information across sensing modalities. The second thrust seeks to optimize system resource utilization for QoE assessment. Multiple personalized models will share common neural network layers at edge servers for resource-efficient model hosting. Adaptive sampling will be applied at VR terminals during sensory data acquisition to maintain data utility. The third thrust focuses on devising data privacy protection mechanisms without sacrificing QoE assessment accuracy. Under the differential privacy framework, the approach features formal quantification of data correlation of multi-modality sensory data, which has been largely overlooked by prior work. The proposed research is inter-disciplinary, which spans data-driven modeling, sensing, and privacy-preserving computing. The proposed mechanisms and designs will be thoroughly evaluated via a combination of measurement campaigns, simulations, and experimental studies.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.