Advanced 3D Video Processing Technologies: Super-Resolution, Chroma-Keying, Depth-Map Generation, and Quality Evaluation

先进的 3D 视频处理技术：超分辨率、色度键控、深度图生成和质量评估

• 批准号: RGPIN-2019-06163
• 负责人: Zhao, Jiying
• 金额: $ 2.04万
• 依托单位: University of Ottawa
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2021
• 资助国家: 加拿大
• 起止时间: 2021-01-01 至 2022-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=738750
• 关键词: Advanced; 3D; Video; Processing; Technologies

Due to the development of electronic technologies, high definition TV and broadcasting have become common in most countries. All the needed facilities and hardware are there ready for 3D and high-resolution videos, but we lack 3D and high-resolution video content. Most existing video content is standard definition video. It would be extremely important, practical, and cost-effective if we could convert existing standard resolution video to high and ultra-high resolution video. Super-resolution (SR) is the technique to generate a high-resolution (HR) image/video with sharp and clear details from its original low-resolution (LR) image/video. In this proposed program, we will research on advanced algorithms on 2D/3D video super-resolution. The chroma-keying is the most widely used compositing technique in photography processing, film making, and television production. Seemingly easy, it is actually extremely difficult to generate satisfactory keying results in different types of studios and for different application setups, due to the uneven background, shading, lighting, transparency, reflection, etc. To date, there are still many difficult issues that need to be solved for 2D image/video keying, and very little has been done to 3D video keying in both academia and industry. Based on the knowledge and experience we have obtained in 2D keying, in this program we will research and develop advanced 3D video keying algorithms. In the proposed research, depth-map generation from 2D and 3D video will be studied, which will enable 2D/3D video conversion and color-depth 3D representation, and provide vital information to chroma-keying and super-resolution, The quality evaluation of super-resolved HR image/video, the chroma-keying, depth-maps, and 3D video will also be studied. Chroma-keying and depth-map generation are mutually dependent. Super-resolution (SR) improves both chroma-keying and depth-map generation; in turn chroma-keying and depth-map can guide super-resolution. Quality evaluation of super-resolution, chroma-keying, depth-map and 3D video is critically important. To respect their own importance and wide applications, we clearly enumerate the four enabling components of 3D video processing in the title: super-resolution, chroma-keying, depth-map generation, and quality evaluation. Just in recent several years, the rapid development of neural network techniques has provided new possibilities for many research fields such as super-resolution, alpha-matting, depth-map generation, and video quality evaluation. In this research program, in addition to conventional techniques, we will also research and develop deep learning based techniques. The proposed research will definitely benefit the academia and industry in autonomous systems & robotics, video broadcasting, film making, image segmentation, object extraction, HDTV, remote sensing, medical imaging, 3D video rendering & coding, etc.

由于电子技术的发展，高清电视和广播在大多数国家都很普遍。所有所需的设施和硬件都准备好用于3D和高分辨率视频，但我们缺乏3D和高分辨率的视频内容。大多数现有视频内容是标准定义视频。如果我们可以将现有的标准分辨率视频转换为高和超高分辨率的视频，那将是极其重要，实用和成本效益的。超分辨率（SR）是生成高分辨率（HR）图像/视频的技术，其原始低分辨率（LR）图像/视频具有清晰明确的细节。在该建议的计划中，我们将在2D/3D视频超分辨率上研究高级算法。 色度钥匙是摄影处理，电影制作和电视制作中最广泛使用的合成技术。似乎很容易，实际上很难在不同类型的工作室和不同的应用程序设置中产生令人满意的钥匙结果，这是由于背景不平，阴影，照明，透明度，反思等。迄今为止，仍然需要为2D映像/视频键起解决许多困难的问题，而对于3D视频键合，在Academia和Academia中都需要解决3D视频键盘。根据我们在2D键合中获得的知识和经验，在此程序中，我们将研究和开发高级3D视频键合算法。 在拟议的研究中，将研究来自2D和3D视频的深度图生成，这将启用2D/3D视频转换和颜色深度3D表示，并为Chroma-Keying和Super-Losolution提供重要信息，超级解决的HR Image/视频的超级分辨HR Image/视频的超级分辨HR Image/the Chorma-Keyinger，Depth-Imaps以及3D的视频也将进行。色度钥匙和深度图的产生是相互依赖的。超分辨率（SR）改善了色度键和深度图的产生；反过来，色度键和深度图可以指导超分辨率。超分辨率，色度键，深度图和3D视频的质量评估至关重要。为了尊重自己的重要性和广泛的应用，我们清楚地列举了标题中3D视频处理的四个启用组件：超分辨率，色度钥匙，深度图生成和质量评估。近几年来，神经网络技术的快速发展为许多研究领域（例如超分辨率，α-麦芽效果，深度图生成和视频质量评估）提供了新的可能性。在该研究计划中，除了传统的技术外，我们还将研究和开发基于深度学习的技术。 拟议的研究肯定会使自主系统和机器人技术，视频广播，胶卷，图像细分，对象提取，HDTV，遥感，医学成像，3D视频渲染和编码等中的学术界和行业受益。