Image Analysis for Realistic Scene Manipulation

现实生活中有许多照片或场景可以在人类中引发强烈的认知反应，比如敬畏或恐惧，或者特定的情绪，比如平静或混乱。视觉刺激和认知过程之间的这种错综复杂的联系使摄影和电影成为艺术表达的重要场所。这种联系也使旨在理解诱导情感和艺术表达的图像分析和综合成为一个很有前途的研究方向，可以在人工智能的背景下开始概念化这种高级认知过程。 目前的计算机视觉和计算机图形系统的能力还不足以让我们开始观察场景的哪些方面使其具有情感意义。这项为期5年的研究计划的主要目标是开发图像分析和处理方法，使人们能够探索电影制作和摄影中艺术选择的认知效果。电影制作和摄影被选为主要的应用对象，因为几十年来发展起来的传统是有效地在视觉上传达思想和情感。这种既定的视觉语言为沟通艺术选择的认知效果和对意象的计算分析提供了一个方便的起点。 拟议的研究计划为三个推进。前两个推力旨在生成场景表示，以隔离构成图像外观的特定方面：(I)包括照明和材料外观在内的低级物理线索，以及(Ii)包括语义和深度在内的高级结构性线索。这两个研究方向都以创新使用机器学习技术为特色，以扩大其在计算摄影中的适用性，旨在通过通用图像表示和交互工具对场景进行逼真和快速的操作。第三个研究重点建立在前两个基础上，将场景处理方法应用于电影后期制作，以分析有助于艺术选择意义的场景的各个方面。 源于物理上逼真的图像表示的毫不费力的图像处理在许多领域都具有很高的价值。拟议的技术将使深度学习在计算摄影中的新应用场景成为可能，并能够扩展现有的计算机视觉数据集，以获得更有效的数据驱动方法。我们的互动编辑工具将公之于众，让独立艺术家和低成本电影更有效地实现他们的艺术愿景。先进的场景剪辑目前只有在加拿大大型电影制作社区拥有大量制作人员和预算的情况下才能实现，这可能会启动电影制作和讲故事的艺术形式的有影响力的创新。

There are many photographs or scenes in real life that can induce strong cognitive responses in humans such as emotions like awe or fear, or specific moods such as calmness or chaos. This intricate connection between visual stimuli and cognitive processes makes photography and movie making important venues for artistic expression. This connection also makes image analysis and synthesis that aims at understanding induced emotions and artistic expression a promising research direction to start conceptualizing such high-level cognitive processes in the context of artificial intelligence. The capabilities of current computer vision and computer graphics systems are not at a point that allows us to start looking at which aspects of a scene make it emotionally significant. The main goal of this 5-year research plan is to develop image analysis and manipulation methods that will enable exploring the cognitive effects of artistic choices in film production and photography. Film production and photography are selected as the main application targets because of the decades of traditions developed to effectively convey ideas and emotions visually. This established visual language provides a convenient starting point for bridging the cognitive effects of artistic choices with computational analysis of imagery. The proposed research are planned as three thrusts. The first two thrusts aim to generate scene representations that isolate specific aspects that contribute to the appearance of imagery: (i) Low-level, physical cues including illumination and material appearance, and (ii) High-level, structural cues including semantics and depth. Both research directions feature innovative uses of machine learning techniques to extend their applicability in computational photography, aiming realistic and quick manipulation of scenes through generic image representations as well as interactive tools. The third research thrust builds on the first two to apply scene manipulation methods to movie post-production to analyze the aspects of a scene that contribute to the significance of artistic choices. Effortless image manipulation that stems from physically realistic image representations is highly valuable in many fields. The proposed techniques will enable new application scenarios of deep learning in computational photography as well as the ability to extend existing computer vision datasets for more effective data-driven methods. Our interactive editing tools will be made publicly available to allow independent artists and low-budget films to achieve their artistic vision more effectively. Making advanced scene editing that is currently only possible with large production crews and budgets available to the large film production community in Canada has the potential to kick off impactful innovations in the art form of movie making and storytelling.