Adaptive Delivery and Analysis of Mixed Reality Content

计算、传感和显示技术的最新进展使新型多媒体内容成为可能，例如360度视频、虚拟现实（VR）和增强现实（AR）以及高光谱视频；我们将它们统称为混合现实（MR）内容。预计MR内容和系统将改变我们日常生活的许多方面，从娱乐和培训服务到工业，医疗和军事应用。然而，实现它们的全部潜力需要解决几个研究挑战。例如，MR内容需要传输大量的数据，并且必须支持用户对内容各个部分的交互性和导航。鉴于互联网流量的持续增长以及其中大部分承载多媒体内容，大量的多媒体流量将需要有效地传递给众多、不同的固定和移动用户。我们通过设计可扩展和自适应流算法来解决这一挑战，这些算法：(i)以最小的带宽开销支持用户交互，（ii）利用现代骨干网的特性（例如，使用软件定义网络（SDN）概念管理网络资源的灵活性），以及（iii）充分利用下一代蜂窝网络特性（例如，虚拟化基站，异构单元和多播支持）。我们将设计算法来优化用户体验质量，减少网络负载，最重要的是，最大限度地减少电池供电的移动设备的能耗。此外，为了使MR内容有用，特别是在军事和工业应用中，更好地理解这些功能丰富的内容是必不可少的。我们将设计分析和放大MR内容中隐藏信息的方法，特别是高光谱视频。高光谱视频比传统视频提供了更多的信息，因为它们捕获的场景跨越许多波段，而不仅仅是可见光范围。例如，它们可以测量远程物体的温度并确定其物质组成。我们将设计对高光谱视频中不同波长波段的信号进行放大的方法，并根据应用需求和可用带宽对复杂视频的各个组成部分进行优先级传输。该研究项目将设计算法和系统，以实现下一代沉浸式多媒体服务，这将为加拿大人带来巨大的经济和社会效益。将在AR/VR、sdn支持网络、5G蜂窝网络、高光谱视频分析与放大等方面培养多名学生，这些都是加拿大高科技公司高需求的前沿和活跃的研究课题。

Recent advances in computing, sensing, and display technologies have enabled new types of multimedia content such as 360-degree videos, virtual reality (VR) and augmented reality (AR), and hyperspectral videos; we collectively refer to them as Mixed Reality (MR) content. MR content and systems are expected to change many aspects of our everyday life, from entertainment and training services to industrial, medical and military applications. Realizing their full potential, however, requires addressing several research challenges. For example, MR content requires transferring substantial amounts of data, and must support users' interactivity and navigation of various parts of the content. Given the continuing growth of the Internet traffic and that most of it carries multimedia content, a vast amount of multimedia traffic will need to be efficiently delivered to numerous, diverse, stationary and mobile users. We address this challenge by designing scalable and adaptive streaming algorithms that: (i) support user interactivities with minimal bandwidth overheads, (ii) leverage the characteristics of modern backbone networks (e.g., flexibility of managing the network resources using software-defined networking (SDN) concepts), and (iii) enable full utilization of next-generation cellular networks features (e.g., virtualized base stations, heterogeneous cells, and multicast support). We will design algorithms to optimize the quality of experience for users, reduce the load on the network, and crucially, minimize the energy consumption of battery-powered mobile devices. In addition, for MR content to be useful especially in military and industrial applications, a better understanding of such feature-rich content is essential. We will design methods to analyze and magnify hidden information in MR content, especially hyperspectral videos. Hyperspectral videos provide substantially more information than traditional videos, because they capture a scene across many wavelength bands, not only in the visible light range as current videos. They can, for example, measure the temperature of a remote object and identify its material composition. We will design methods to magnify the signals in different wavelength bands in hyperspectral videos, and to prioritize the transmission of various components of such complex videos based on the application needs and available bandwidth. ***This research program will design algorithms and systems to enable next-generation immersive multimedia services, which will potentially have substantial economic and social benefits for Canadians. It will train several students in the theoretical and practical aspects of AR/VR, SDN-enabled networks, 5G cellular networks, and analysis and magnification of hyperspectral videos, which are all cutting-edge and active research topics of high demand by Canadian high-tech companies.