Image Analysis for Realistic Scene Manipulation

点击翻译按钮获取中文摘要

There are many photographs or scenes in real life that can induce strong cognitive responses in humans such as emotions like awe or fear, or specific moods such as calmness or chaos. This intricate connection between visual stimuli and cognitive processes makes photography and movie making important venues for artistic expression. This connection also makes image analysis and synthesis that aims at understanding induced emotions and artistic expression a promising research direction to start conceptualizing such high-level cognitive processes in the context of artificial intelligence. The capabilities of current computer vision and computer graphics systems are not at a point that allows us to start looking at which aspects of a scene make it emotionally significant. The main goal of this 5-year research plan is to develop image analysis and manipulation methods that will enable exploring the cognitive effects of artistic choices in film production and photography. Film production and photography are selected as the main application targets because of the decades of traditions developed to effectively convey ideas and emotions visually. This established visual language provides a convenient starting point for bridging the cognitive effects of artistic choices with computational analysis of imagery. The proposed research are planned as three thrusts. The first two thrusts aim to generate scene representations that isolate specific aspects that contribute to the appearance of imagery: (i) Low-level, physical cues including illumination and material appearance, and (ii) High-level, structural cues including semantics and depth. Both research directions feature innovative uses of machine learning techniques to extend their applicability in computational photography, aiming realistic and quick manipulation of scenes through generic image representations as well as interactive tools. The third research thrust builds on the first two to apply scene manipulation methods to movie post-production to analyze the aspects of a scene that contribute to the significance of artistic choices. Effortless image manipulation that stems from physically realistic image representations is highly valuable in many fields. The proposed techniques will enable new application scenarios of deep learning in computational photography as well as the ability to extend existing computer vision datasets for more effective data-driven methods. Our interactive editing tools will be made publicly available to allow independent artists and low-budget films to achieve their artistic vision more effectively. Making advanced scene editing that is currently only possible with large production crews and budgets available to the large film production community in Canada has the potential to kick off impactful innovations in the art form of movie making and storytelling.