Enabling technologies of the video systems of future

最近，一系列史无前例的、不同种类的新成像设备，如图像+深度传感器、光场相机和全息设备，使新的捕捉和可视化视角成为可能。这推动了视觉内容获取和消费的颠覆性范式转变：从平面、2D表示转向体积和沉浸式模式的成像。为了能够实现这些新颖的表示，已经开发了一种被称为“全光”的数学框架，该框架考虑场景内任何点的通常颜色信息，并添加关于从不同位置观察时颜色变化的方向信息。在光场、全息图和点云等全光模式普及之前，仍有许多科学挑战需要研究人员解决。从获取的角度来看，全光成像需要捕获大量高维数据的能力。一方面，这种视觉数据的高维表示提供了强大的能力，并具有大幅提高传统计算机视觉任务的性能的潜力。另一方面，全光的高维性也给数据捕获、数据压缩、内容编辑和显示带来了新的挑战。该研究计划致力于开发不同全光模式的采集、压缩、通信和显示技术。在短期内，我们将探索全息和光场数据的可伸缩编码，这将有助于从码流中提取不同级别的质量、空间分辨率、深度分辨率和视角数量。我们还将研究光场视频的帧速率上转换以及光场成像在目标跟踪中的应用。本研究计划中提出的适当压缩和处理技术的开发将使全光模式成为可能并增加其普及度。这项研究产生的新知识将进入对加拿大非常重要的领域的新技术和产品，如医疗保健、教育和汽车工业。

Recently, an unprecedented and heterogeneous range of new imaging devices such as image+depth sensors, light-field cameras and holographic devices have enabled new capture and visualization perspectives. This is driving a disruptive paradigm shift in the acquisition and consumption of visual content: moving from a planar, 2D representation towards imaging in volumetric and immersive modalities. To enable these novel representations, a mathematical framework called ``plenoptic" that considers the usual color information of any point within a scene and adds directional information about the colour change when observed from different positions has been developed. Many scientific challenges remain for researchers to address before plenoptic modalities such as light field, hologram and point cloud become prevalent. From an acquisition point of view, plenoptic imaging requires the ability to capture a large amount of high dimensional data. On the one hand, this higher dimensional representation of visual data offers powerful capabilities and has the potential to substantially improve the performance of traditional computer vision tasks. On the other hand, the high-dimensionality of plenoptic also brings up new challenges in terms of data capture, data compression, content editing, and display. This research program seeks to develop technologies for acquisition, compression, communication and display of different plenoptic modalities. In the short term we will explore scalable coding of holographic and light field data which will facilitate the extraction of different levels of quality, spatial resolution, depth resolution and number of viewing angles from the codestream. We will also investigate frame rate up conversion of light field video and the application of light field imaging for object tracking. The development of proper compression and processing technologies proposed in this research program will enable and increase the popularity of plenoptic modalities. The new knowledge arising from this research will find its way into new technologies and products in areas of great importance to Canada such as healthcare, education and auto industry.