Image Analysis for Realistic Scene Manipulation

在真实的生活中，有许多照片或场景可以引起人类强烈的认知反应，如敬畏或恐惧等情绪，或平静或混乱等特定情绪。视觉刺激和认知过程之间的这种错综复杂的联系使摄影和电影制作成为艺术表达的重要场所。这种联系也使得旨在理解诱导情感和艺术表达的图像分析和合成成为一个有前途的研究方向，可以在人工智能的背景下开始概念化这种高级认知过程。 目前的计算机视觉和计算机图形系统的能力还不足以让我们开始研究场景的哪些方面使其具有情感意义。这项为期5年的研究计划的主要目标是开发图像分析和操作方法，以探索电影制作和摄影中艺术选择的认知效果。电影制作和摄影被选为主要的应用目标，因为几十年的传统发展，以有效地传达思想和情感的视觉。这种既定的视觉语言提供了一个方便的起点，桥接艺术选择的认知效果与图像的计算分析。 拟议的研究计划分为三个重点。前两个推力的目的是生成场景表示，隔离的具体方面，有助于图像的外观：（i）低级别，物理线索，包括照明和材料的外观，和（ii）高级别，结构线索，包括语义和深度。这两个研究方向都以机器学习技术的创新应用为特色，以扩展其在计算摄影中的适用性，旨在通过通用图像表示和交互式工具对场景进行逼真和快速的操作。第三个研究重点是在前两个基础上，将场景操作方法应用于电影后期制作，分析场景的各个方面对艺术选择的意义。 从物理上真实的图像表示出发，进行简单的图像处理在许多领域都具有很高的价值。所提出的技术将使深度学习在计算摄影中的新应用场景成为可能，并能够扩展现有的计算机视觉数据集，以获得更有效的数据驱动方法。我们的互动编辑工具将向公众开放，让独立艺术家和低成本电影更有效地实现他们的艺术愿景。制作先进的场景编辑，目前只有大型制作团队和预算才能提供给加拿大的大型电影制作社区，有可能在电影制作和讲故事的艺术形式中启动有影响力的创新。

There are many photographs or scenes in real life that can induce strong cognitive responses in humans such as emotions like awe or fear, or specific moods such as calmness or chaos. This intricate connection between visual stimuli and cognitive processes makes photography and movie making important venues for artistic expression. This connection also makes image analysis and synthesis that aims at understanding induced emotions and artistic expression a promising research direction to start conceptualizing such high-level cognitive processes in the context of artificial intelligence. The capabilities of current computer vision and computer graphics systems are not at a point that allows us to start looking at which aspects of a scene make it emotionally significant. The main goal of this 5-year research plan is to develop image analysis and manipulation methods that will enable exploring the cognitive effects of artistic choices in film production and photography. Film production and photography are selected as the main application targets because of the decades of traditions developed to effectively convey ideas and emotions visually. This established visual language provides a convenient starting point for bridging the cognitive effects of artistic choices with computational analysis of imagery. The proposed research are planned as three thrusts. The first two thrusts aim to generate scene representations that isolate specific aspects that contribute to the appearance of imagery: (i) Low-level, physical cues including illumination and material appearance, and (ii) High-level, structural cues including semantics and depth. Both research directions feature innovative uses of machine learning techniques to extend their applicability in computational photography, aiming realistic and quick manipulation of scenes through generic image representations as well as interactive tools. The third research thrust builds on the first two to apply scene manipulation methods to movie post-production to analyze the aspects of a scene that contribute to the significance of artistic choices. Effortless image manipulation that stems from physically realistic image representations is highly valuable in many fields. The proposed techniques will enable new application scenarios of deep learning in computational photography as well as the ability to extend existing computer vision datasets for more effective data-driven methods. Our interactive editing tools will be made publicly available to allow independent artists and low-budget films to achieve their artistic vision more effectively. Making advanced scene editing that is currently only possible with large production crews and budgets available to the large film production community in Canada has the potential to kick off impactful innovations in the art form of movie making and storytelling.