Enabling technologies of the video systems of future

最近，一系列前所未有的新型成像设备，如图像+深度传感器、光场相机和全息设备，已经实现了新的捕获和可视化视角。这正在推动视觉内容的获取和消费的颠覆性范式转变：从平面，2D表示转向体积和沉浸式模式的成像。为了实现这些新的表示，一个数学框架称为“全光”，考虑到场景内任何点的通常颜色信息，并添加了从不同位置观察时颜色变化的方向信息。在光场、全息图和点云等全光模态变得普遍之前，研究人员仍面临许多科学挑战。从采集的角度来看，全光成像需要捕获大量高维数据的能力。一方面，这种视觉数据的高维表示提供了强大的功能，并有可能大大提高传统计算机视觉任务的性能。另一方面，全光的高维性也在数据捕获、数据压缩、内容编辑和显示方面带来了新的挑战。该研究计划旨在开发用于不同全光模式的采集，压缩，通信和显示的技术。在短期内，我们将探索全息和光场数据的可扩展编码，这将有助于从码流中提取不同级别的质量，空间分辨率，深度分辨率和视角数量。我们也将研究光场影像的帧频上转换以及光场影像在目标追踪上的应用。在这项研究计划中提出的适当的压缩和处理技术的发展将使全光模式的普及。从这项研究中产生的新知识将在对加拿大非常重要的领域，如医疗保健，教育和汽车工业的新技术和产品中找到出路。

Recently, an unprecedented and heterogeneous range of new imaging devices such as image+depth sensors, light-field cameras and holographic devices have enabled new capture and visualization perspectives. This is driving a disruptive paradigm shift in the acquisition and consumption of visual content: moving from a planar, 2D representation towards imaging in volumetric and immersive modalities. To enable these novel representations, a mathematical framework called ``plenoptic" that considers the usual color information of any point within a scene and adds directional information about the colour change when observed from different positions has been developed. Many scientific challenges remain for researchers to address before plenoptic modalities such as light field, hologram and point cloud become prevalent. From an acquisition point of view, plenoptic imaging requires the ability to capture a large amount of high dimensional data. On the one hand, this higher dimensional representation of visual data offers powerful capabilities and has the potential to substantially improve the performance of traditional computer vision tasks. On the other hand, the high-dimensionality of plenoptic also brings up new challenges in terms of data capture, data compression, content editing, and display. This research program seeks to develop technologies for acquisition, compression, communication and display of different plenoptic modalities. In the short term we will explore scalable coding of holographic and light field data which will facilitate the extraction of different levels of quality, spatial resolution, depth resolution and number of viewing angles from the codestream. We will also investigate frame rate up conversion of light field video and the application of light field imaging for object tracking. The development of proper compression and processing technologies proposed in this research program will enable and increase the popularity of plenoptic modalities. The new knowledge arising from this research will find its way into new technologies and products in areas of great importance to Canada such as healthcare, education and auto industry.