Advanced 3D Video Processing Technologies: Super-Resolution, Chroma-Keying, Depth-Map Generation, and Quality Evaluation

由于电子技术的发展，高清晰度电视和广播在大多数国家已经变得普遍。3D和高分辨率视频所需的所有设施和硬件都已准备就绪，但我们缺乏3D和高分辨率视频内容。大多数现有的视频内容都是标清视频。如果我们能够将现有的标准分辨率视频转换为高分辨率和超高分辨率视频，这将是非常重要、实用和经济的。超分辨率（SR）是一种将原始的低分辨率（LR）图像/视频生成具有清晰细节的高分辨率（HR）图像/视频的技术。在本计划中，我们将研究2D/3D视频超分辨率的先进算法。色键法是在摄影处理、电影制作和电视制作中应用最广泛的合成技术。看起来很简单，实际上在不同类型的工作室和不同的应用程序设置中产生令人满意的关键结果是非常困难的，因为不均匀的背景，阴影，照明，透明度，反射等。迄今为止，2D图像/视频键控仍然有许多难题需要解决，而3D视频键控在学术界和工业界都做得很少。基于我们在2D键控方面获得的知识和经验，在这个项目中我们将研究和开发先进的3D视频键控算法。在本研究中，将研究2D和3D视频的深度图生成，这将使2D/3D视频转换和彩色深度3D表示成为可能，并为彩色键控和超分辨率提供重要信息。此外，还将研究超分辨率HR图像/视频的质量评估、彩色键控、深度图和3D视频。色度键控和深度图生成是相互依赖的。超分辨率（SR）改进了色键和深度图的生成；反过来，色键和深度图可以引导超分辨率。超分辨率、彩色键控、深度图和3D视频的质量评价至关重要。为了尊重它们自身的重要性和广泛的应用，我们在标题中清楚地列举了3D视频处理的四个使能组件：超分辨率、色键控、深度图生成和质量评估。近年来，神经网络技术的快速发展为超分辨率、阿尔法抠图、深度图生成、视频质量评价等研究领域提供了新的可能性。在这个研究项目中，除了传统技术外，我们还将研究和开发基于深度学习的技术。该研究将在自主系统和机器人、视频广播、电影制作、图像分割、目标提取、高清电视、遥感、医学成像、3D视频渲染和编码等领域为学术界和工业界带来益处。

Due to the development of electronic technologies, high definition TV and broadcasting have become common in most countries. All the needed facilities and hardware are there ready for 3D and high-resolution videos, but we lack 3D and high-resolution video content. Most existing video content is standard definition video. It would be extremely important, practical, and cost-effective if we could convert existing standard resolution video to high and ultra-high resolution video. Super-resolution (SR) is the technique to generate a high-resolution (HR) image/video with sharp and clear details from its original low-resolution (LR) image/video. In this proposed program, we will research on advanced algorithms on 2D/3D video super-resolution. The chroma-keying is the most widely used compositing technique in photography processing, film making, and television production. Seemingly easy, it is actually extremely difficult to generate satisfactory keying results in different types of studios and for different application setups, due to the uneven background, shading, lighting, transparency, reflection, etc. To date, there are still many difficult issues that need to be solved for 2D image/video keying, and very little has been done to 3D video keying in both academia and industry. Based on the knowledge and experience we have obtained in 2D keying, in this program we will research and develop advanced 3D video keying algorithms. In the proposed research, depth-map generation from 2D and 3D video will be studied, which will enable 2D/3D video conversion and color-depth 3D representation, and provide vital information to chroma-keying and super-resolution, The quality evaluation of super-resolved HR image/video, the chroma-keying, depth-maps, and 3D video will also be studied. Chroma-keying and depth-map generation are mutually dependent. Super-resolution (SR) improves both chroma-keying and depth-map generation; in turn chroma-keying and depth-map can guide super-resolution. Quality evaluation of super-resolution, chroma-keying, depth-map and 3D video is critically important. To respect their own importance and wide applications, we clearly enumerate the four enabling components of 3D video processing in the title: super-resolution, chroma-keying, depth-map generation, and quality evaluation. Just in recent several years, the rapid development of neural network techniques has provided new possibilities for many research fields such as super-resolution, alpha-matting, depth-map generation, and video quality evaluation. In this research program, in addition to conventional techniques, we will also research and develop deep learning based techniques. The proposed research will definitely benefit the academia and industry in autonomous systems & robotics, video broadcasting, film making, image segmentation, object extraction, HDTV, remote sensing, medical imaging, 3D video rendering & coding, etc.