Image Analysis for Realistic Scene Manipulation

现实生活中有许多照片或场景可以引起人类强烈的认知反应，如敬畏或恐惧等情绪，或平静或混乱等特定情绪。这种视觉刺激与认知过程之间的复杂联系使得摄影和电影制作成为艺术表达的重要场所。这种联系也使得旨在理解诱发情绪和艺术表现的图像分析与合成成为在人工智能背景下开始概念化这种高级认知过程的一个有前途的研究方向。当前的计算机视觉和计算机图形系统的能力还不能让我们开始关注场景的哪些方面使其具有情感意义。这个五年研究计划的主要目标是发展图像分析和处理方法，以探索电影制作和摄影中艺术选择的认知效果。之所以选择电影制作和摄影作为主要的应用对象，是因为几十年来形成的传统在视觉上有效地传达思想和情感。这种既定的视觉语言为连接艺术选择的认知效果与图像的计算分析提供了一个方便的起点。拟议的研究计划分为三个重点。前两个重点旨在生成场景表征，分离出有助于图像外观的特定方面：(i)低水平的物理线索，包括照明和材料外观；（ii）高水平的结构线索，包括语义和深度。这两个研究方向都具有机器学习技术的创新用途，以扩展其在计算摄影中的适用性，旨在通过通用图像表示和交互工具实现场景的逼真和快速操作。第三个研究重点建立在前两个的基础上，将场景处理方法应用于电影后期制作，以分析场景中有助于艺术选择意义的各个方面。毫不费力的图像处理源于物理逼真的图像表示，在许多领域具有很高的价值。所提出的技术将在计算摄影中实现深度学习的新应用场景，以及扩展现有计算机视觉数据集的能力，以获得更有效的数据驱动方法。我们的互动编辑工具将向公众开放，让独立艺术家和低成本电影更有效地实现他们的艺术愿景。目前，只有在加拿大大型电影制作社区拥有庞大的制作团队和预算的情况下，才能进行高级场景编辑，这有可能在电影制作和讲故事的艺术形式中开启有影响力的创新。

There are many photographs or scenes in real life that can induce strong cognitive responses in humans such as emotions like awe or fear, or specific moods such as calmness or chaos. This intricate connection between visual stimuli and cognitive processes makes photography and movie making important venues for artistic expression. This connection also makes image analysis and synthesis that aims at understanding induced emotions and artistic expression a promising research direction to start conceptualizing such high-level cognitive processes in the context of artificial intelligence. The capabilities of current computer vision and computer graphics systems are not at a point that allows us to start looking at which aspects of a scene make it emotionally significant. The main goal of this 5-year research plan is to develop image analysis and manipulation methods that will enable exploring the cognitive effects of artistic choices in film production and photography. Film production and photography are selected as the main application targets because of the decades of traditions developed to effectively convey ideas and emotions visually. This established visual language provides a convenient starting point for bridging the cognitive effects of artistic choices with computational analysis of imagery. The proposed research are planned as three thrusts. The first two thrusts aim to generate scene representations that isolate specific aspects that contribute to the appearance of imagery: (i) Low-level, physical cues including illumination and material appearance, and (ii) High-level, structural cues including semantics and depth. Both research directions feature innovative uses of machine learning techniques to extend their applicability in computational photography, aiming realistic and quick manipulation of scenes through generic image representations as well as interactive tools. The third research thrust builds on the first two to apply scene manipulation methods to movie post-production to analyze the aspects of a scene that contribute to the significance of artistic choices. Effortless image manipulation that stems from physically realistic image representations is highly valuable in many fields. The proposed techniques will enable new application scenarios of deep learning in computational photography as well as the ability to extend existing computer vision datasets for more effective data-driven methods. Our interactive editing tools will be made publicly available to allow independent artists and low-budget films to achieve their artistic vision more effectively. Making advanced scene editing that is currently only possible with large production crews and budgets available to the large film production community in Canada has the potential to kick off impactful innovations in the art form of movie making and storytelling.