CAREER: Art and Vision: Scene Layout from Pictorial Cues

职业：艺术与视觉：图画线索的场景布局PI：Stella (XingXing) Yu 院校：波士顿学院艺术家是视觉感知的大师。一起研究艺术和视觉可以为计算机视觉的基本问题提供新的解决方案。我们专注于从单个图像推断场景布局。这个问题从人工智能研究的早期就开始研究，产生了许多所谓的“Shape-from-X”方法，其中 X 可以是阴影、透视等。不幸的是，这些方法中的每一种都在其自己的假设下工作，而这些假设通常在真实图像中并不成立。这些线索如何相互作用和整合仍然难以捉摸。画家经常结合使用四种技术：遮挡、透视、阴影和形式，以有效地从 2D 图像中唤起 3D 感知。研究他们的技术可以深入了解从像素值恢复场景布局的计算。 PI 提议将艺术家和视觉科学家聚集在一起，解决根据图片线索进行场景布局的计算问题。该项目通过三个领域实现这一目标：教育、实验和计算建模。波士顿学院开发了新的跨学科课程“艺术与视觉感知”，以全面交叉检验艺术如何有助于对视觉的理解，以及视觉如何有助于艺术的生成和观看。学生们积极参与艺术实践和视觉实验。一起学习艺术和视觉比单独学习每个学科有更深入的理解。学生的作业还为视觉研究提供了有价值的数据集。在该项目中，根据图形线索进行场景布局的计算方法是根据光谱图论框架中多个图形线索的全局集成，将像素分组到空间组织的表面中。目标是将有关这些线索如何相互作用的艺术渲染知识转化为计算现实。PI 将使用眼动追踪、心理物理学实验和计算模型来研究几何（遮挡和透视）、外观（亮度和颜色）和形式。这些工作分为两个阶段，从从由平面（房间和街道）构成的场景推断空间布局到由曲面（景观和一般场景）构成的场景推断空间布局。智力优点视觉最引人注目的是它能够从单个 2D 图像感知 3D 空间布局。所提出的研究从分组的角度复制了这种计算能力。与统计学习方法相比，分组方法不仅是通用的，因此可以很好地适应场景的数量，而且还可以对场景中的表面进行精确的组织。与传统的 Shape-from-X 方法相比，分组方法将每个图形线索与其他图形线索结合起来检查。这些多种图形线索的整合使它们首次适用于真实图像。 PI 开发了谱图理论中用于深度分离的基本分组机制。与该主题的大多数现有公式相比，它具有无与伦比的概念简单性、计算效率并保证近乎全局最优性。拟议的亮度和颜色感知研究，与形状从阴影和表面组织相结合，将有助于澄清低级和高级机制在赫林和亥姆霍兹之间关于颜色感知的长期科学争论中的作用。更广泛的影响该项目通过开发横跨神经科学、心理学、计算机科学和视觉艺术领域的新课程，让学生参与艺术实践和科学实验，并提供一个论坛，不仅在研究方面而且在教育方面弥合了艺术与科学之间的差距。艺术家和科学家交流视觉感知的想法。这些跨学科的努力符合波士顿学院的文科教育传统。该项目不仅受益于校园内强大的美术系，还将培养非技术人员的计算机科学意识和外展，促进波士顿学院年轻的计算机科学系的成长。网址：http://www.cs.bc.edu/~syu/artvis/

CAREER: Art and Vision: Scene Layout from Pictorial CuesPI: Stella (XingXing) Yu Institution: Boston CollegeArtists are the masters of visual perception. Studying art and vision together can provide new solutions to fundamental problems in computer vision. We focus on inferring scene layout from a single image. This problem has been studied since the earliest days of Artificial Intelligence research, resulting in a host of so-called Shape-from-X methods, where X could be shading, perspective, etc.Unfortunately, each of these methods works under its own assumptions which often do not hold in real images. How these cues interact and integrate remains elusive. Painters constantly use a combination of four techniques: occlusion, perspective, shading, and form to effectively evoke a 3D percept from a 2D picture. Studying their techniques can lend insights into the computation of recovering scene layout from pixel values. The PI proposes to bring artists and vision scientists together to solve the computational problem of scene layout from pictorial cues. This project realizes it in three areas:education, experiments and computational modeling.A new interdisciplinary course, Art and Visual Perception, has been developed at Boston College to give a comprehensive cross-examination of how art contributes to the understanding of vision, and how vision contributes to the generation and viewing of art. Students are actively engaged in both art practice and vision experiments.Learning art and vision together results in a deeper understanding than studying each discipline separately. Students' assignments also result in valuable datasets for vision research.The computational approach to scene layout from pictorial cues in this project is to group pixels into spatially organized surfaces from a global integration of multiple pictorial cues in a spectral graph-theoretic framework. The goal is to turn artistic rendering knowledge on how these cues interact into a computational reality.The PI will study geometry (occlusion and perspective), appearance (brightness and color), and form using eye tracking and psychophysics experiments and computational models. These efforts are organized into two phases that progress from inferring the spatial layout from scenes made of planar surfaces (rooms and streets) to scenes made of curved surfaces (landscape and generic scenes).Intellectual MeritWhat is most remarkable about vision is its ability to perceive 3D spatial layout from a single 2D image. The proposed research replicates this ability in computation from a grouping perspective.Compared to statistical learning approaches, the grouping method is not only generic and thus scales well with the number of scenes, but can also produce a precise organization of surfaces in the scene.Compared to traditional Shape-from-X approaches, the grouping method examines each pictorial cue in conjunction with others. The integration of these multiple pictorial cues allows them for the first time to become applicable to real images. The PI has developed the essential grouping machinery in spectral graph theory for depth segregation. Compared to most existing formulations on this topic, it has unparalleled conceptual simplicity, computational efficiency, and guaranteed near-global optimality. The proposed research on brightness and color perception, in connection with Shape-from- Shading and surface organization, will help clarify the role of low- level and high-level mechanisms in the long-standing scientific debate between Hering and Helmholtz on color perception.Broader ImpactThis project bridges the gap between art and science not only in research but also in education by developing a new curriculum that traverses the areas of neuroscience, psychology, computer science, and visual arts, by involving students in art practice and scientific experiments, and by providing a forum for artists and scientists to exchange ideas on visual perception. These interdisciplinary efforts befit the liberal arts education tradition at Boston College. This project will not only benefit from the strong Fine Arts department on campus, but also cultivate computer science awareness and outreach to non-technical people, and promote the growth of the young Computer Science department at Boston College.URL: http://www.cs.bc.edu/~syu/artvis/