Adaptive Delivery and Analysis of Mixed Reality Content

计算、传感和显示技术的最新进展使新型多媒体内容成为可能，例如360度视频、虚拟现实（VR）和增强现实（AR）以及高光谱视频;我们将它们统称为混合现实（MR）内容。MR内容和系统预计将改变我们日常生活的许多方面，从娱乐和培训服务到工业，医疗和军事应用。然而，要充分发挥其潜力，就需要应对若干研究挑战。例如，MR内容需要传输大量的数据，并且必须支持用户对内容的各个部分的交互和导航。考虑到互联网业务的持续增长以及其中大部分承载多媒体内容，大量的多媒体业务将需要被有效地传递给众多的、不同的、固定的和移动的用户。我们通过设计可扩展和自适应的流媒体算法来解决这一挑战，这些算法：（i）以最小的带宽开销支持用户交互，（ii）利用现代骨干网络的特性（例如，使用软件定义网络（SDN）概念管理网络资源的灵活性），以及（iii）能够充分利用下一代蜂窝网络特征（例如，虚拟化基站、异构小区和多播支持）。我们将设计算法来优化用户的体验质量，减少网络负载，最重要的是，最大限度地减少电池供电的移动的设备的能耗。此外，对于MR内容来说，特别是在军事和工业应用中，更好地理解这种功能丰富的内容是必不可少的。我们将设计方法来分析和放大MR内容中隐藏的信息，特别是高光谱视频。高光谱视频比传统视频提供更多的信息，因为它们在许多波长波段上捕捉场景，而不仅仅是在可见光范围内。例如，它们可以测量远程物体的温度并识别其材料成分。我们将设计方法来放大高光谱视频中不同波段的信号，并根据应用需求和可用带宽优先传输此类复杂视频的各种组件。* 这项研究计划将设计算法和系统，以实现下一代沉浸式多媒体服务，这将有可能为加拿大人带来巨大的经济和社会效益。它将在AR/VR，SDN网络，5G蜂窝网络以及高光谱视频的分析和放大的理论和实践方面培养几名学生，这些都是加拿大高科技公司高需求的前沿和活跃的研究课题。

Recent advances in computing, sensing, and display technologies have enabled new types of multimedia content such as 360-degree videos, virtual reality (VR) and augmented reality (AR), and hyperspectral videos; we collectively refer to them as Mixed Reality (MR) content. MR content and systems are expected to change many aspects of our everyday life, from entertainment and training services to industrial, medical and military applications. Realizing their full potential, however, requires addressing several research challenges. For example, MR content requires transferring substantial amounts of data, and must support users' interactivity and navigation of various parts of the content. Given the continuing growth of the Internet traffic and that most of it carries multimedia content, a vast amount of multimedia traffic will need to be efficiently delivered to numerous, diverse, stationary and mobile users. We address this challenge by designing scalable and adaptive streaming algorithms that: (i) support user interactivities with minimal bandwidth overheads, (ii) leverage the characteristics of modern backbone networks (e.g., flexibility of managing the network resources using software-defined networking (SDN) concepts), and (iii) enable full utilization of next-generation cellular networks features (e.g., virtualized base stations, heterogeneous cells, and multicast support). We will design algorithms to optimize the quality of experience for users, reduce the load on the network, and crucially, minimize the energy consumption of battery-powered mobile devices. In addition, for MR content to be useful especially in military and industrial applications, a better understanding of such feature-rich content is essential. We will design methods to analyze and magnify hidden information in MR content, especially hyperspectral videos. Hyperspectral videos provide substantially more information than traditional videos, because they capture a scene across many wavelength bands, not only in the visible light range as current videos. They can, for example, measure the temperature of a remote object and identify its material composition. We will design methods to magnify the signals in different wavelength bands in hyperspectral videos, and to prioritize the transmission of various components of such complex videos based on the application needs and available bandwidth. ***This research program will design algorithms and systems to enable next-generation immersive multimedia services, which will potentially have substantial economic and social benefits for Canadians. It will train several students in the theoretical and practical aspects of AR/VR, SDN-enabled networks, 5G cellular networks, and analysis and magnification of hyperspectral videos, which are all cutting-edge and active research topics of high demand by Canadian high-tech companies.