State of the Art on Neural Rendering
State of the Art on Neural Rendering
复制标题
DOI：
10.1111/cgf.14022
复制
发表时间：
2020-04
影响因子：
2.5
通讯作者：
A. Tewari;Ohad Fried;Justus Thies;V. Sitzmann;Stephen Lombardi;Kalyan Sunkavalli;Ricardo Martin-Brualla;T. Simon;Jason M. Saragih;M. Nießner;Rohit Pandey;S. Fanello;Gordon Wetzstein;Jun-Yan Zhu;C. Theobalt;Maneesh Agrawala;Eli Shechtman;Dan B. Goldman;Michael Zollhofer
中科院分区：
文献类型：
--
作者：
A. Tewari;Ohad Fried;Justus Thies;V. Sitzmann;Stephen Lombardi;Kalyan Sunkavalli;Ricardo Martin-Brualla;T. Simon;Jason M. Saragih;M. Nießner;Rohit Pandey;S. Fanello;Gordon Wetzstein;Jun-Yan Zhu;C. Theobalt;Maneesh Agrawala;Eli Shechtman;Dan B. Goldman;Michael Zollhofer
高效渲染逼真的虚拟世界是计算机图形学的长期努力。现代图形技术已经成功地从手工制作的场景表示中合成照片般逼真的图像。然而，场景的形状、材质、照明和其他方面的自动生成仍然是一个具有挑战性的问题，如果解决了这个问题，将使照片级逼真的计算机图形更广泛地使用。与此同时，计算机视觉和机器学习的进步催生了一种新的图像合成和编辑方法，即深度生成模型。神经渲染是一个快速新兴的新兴领域，它将生成机器学习技术与计算机图形学的物理知识相结合，例如，通过将可微渲染集成到网络训练中。随着计算机图形和视觉领域的大量应用，神经渲染有望成为图形界的一个新领域，但目前还没有对这一新兴领域的调查。这份最先进的报告总结了神经渲染的最新趋势和应用。我们专注于将经典计算机图形技术与深度生成模型相结合的方法，以获得可控且逼真的输出。从底层计算机图形和机器学习概念的概述开始，我们讨论神经渲染方法的关键方面。具体来说，我们的重点是控制的类型，即如何提供控制、学习管道的哪些部分、显式控制与隐式控制、泛化以及随机综合与确定性综合。这份最先进报告的后半部分重点介绍了所描述算法的许多重要用例，例如新颖的视图合成、语义照片处理、面部和身体重演、重新照明、自由视点视频以及为虚拟和增强现实远程呈现创建照片般真实的化身。最后，我们讨论了此类技术的社会影响并调查了开放的研究问题。
Efficient rendering of photo‐realistic virtual worlds is a long standing effort of computer graphics. Modern graphics techniques have succeeded in synthesizing photo‐realistic images from hand‐crafted scene representations. However, the automatic generation of shape, materials, lighting, and other aspects of scenes remains a challenging problem that, if solved, would make photo‐realistic computer graphics more widely accessible. Concurrently, progress in computer vision and machine learning have given rise to a new approach to image synthesis and editing, namely deep generative models. Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e.g., by the integration of differentiable rendering into network training. With a plethora of applications in computer graphics and vision, neural rendering is poised to become a new area in the graphics community, yet no survey of this emerging field exists. This state‐of‐the‐art report summarizes the recent trends and applications of neural rendering. We focus on approaches that combine classic computer graphics techniques with deep generative models to obtain controllable and photorealistic outputs. Starting with an overview of the underlying computer graphics and machine learning concepts, we discuss critical aspects of neural rendering approaches. Specifically, our emphasis is on the type of control, i.e., how the control is provided, which parts of the pipeline are learned, explicit vs. implicit control, generalization, and stochastic vs. deterministic synthesis. The second half of this state‐of‐the‐art report is focused on the many important use cases for the described algorithms such as novel view synthesis, semantic photo manipulation, facial and body reenactment, relighting, free‐viewpoint video, and the creation of photo‐realistic avatars for virtual and augmented reality telepresence. Finally, we conclude with a discussion of the social implications of such technology and investigate open research problems.