RI: Small: Visual Situation Recognition: An Integration of Deep Networks and Analogy-Making

这个项目研究了一种新的方法来构建能够识别视觉情况的计算机系统。虽然计算机视觉的很多努力都集中在识别图像中的孤立物体上，但人们实际上做的是识别连贯的情况——物体的集合及其相互关系，这些集合在一起，对应于一个已知的概念，比如“一个孩子的生日聚会”，或者“一个人在海滩上遛狗”，或者“两个人要打架”，或者“一个盲人过马路”。从表面上看，人类的情景识别似乎毫不费力，但它依赖于人类感知物体的能力、物体之间的关系系统以及与存储的知识和记忆的类比之间复杂的动态相互作用。目前还没有计算机视觉系统接近于捕捉人类的这些能力。使计算机能够灵活地识别视觉情况，将在医学诊断、科学图像解释、增强人机交互和个人信息组织等不同领域创造大量重要应用。本项目探索的方法整合了先前研究的两种方法：用于低级视觉的脑启发神经网络和概念和类比制作的认知水平模型。在这个集成架构中，通过与存储概念结构的类比来识别情况将是一个动态过程，在这个过程中，随着感知的展开，自下而上（感知）和自上而下（概念）的影响相互影响。如果成功，该系统将能够以一种与场景的复杂性和被识别的抽象概念相适应的方式识别视觉情境。作为该项目的一部分，将收集一些反映不同抽象视觉情况的基准图像数据集来评估识别系统。此外，PI将使用收集到的数据集，设计并运行一个关于视觉情况自动识别的公开竞赛。本次竞赛将促进这一主题的研究，并帮助研究人员在这一领域评估各种方法的成功，并衡量抽象视觉识别技术的当前状态。在这个项目中开发的所有源代码和基准数据库将通过网络公开提供。

This project investigates a novel approach to building computer systems that can recognize visual situations. While much effort in computer vision has focused on identifying isolated objects in images, what people actually do is recognize coherent situations--collections of objects and their interrelations that, taken together, correspond to a known concept, such as "a child's birthday party," or "a man walking a dog on the beach," or "two people about to fight," or "a blind person crossing the street." Situation recognition by humans may appear on the surface to be effortless, but it relies on a complex dynamic interplay among human abilities to perceive objects, systems of relationships among objects, and analogies with stored knowledge and memories. No computer vision system yet comes close to capturing these human abilities. Enabling computers to flexibly recognize visual situations would create a flood of important applications in fields as diverse as medical diagnosis, interpretation of scientific imagery, enhanced human-computer interaction, and personal information organization.The approach explored in this project integrates two previously studied approaches: brain-inspired neural networks for lower-level vision and cognitive-level models of concepts and analogy-making. In this integrated architecture, recognizing situations--via analogies with stored conceptual structures--will be a dynamic process in which bottom-up (perceptual) and top-down (conceptual) influences affect one another as perception unfolds. If successful, this system will be able to recognize visual situations in a way that scales well with the complexity of the scene and the abstract concept being recognized. As part of this project, a number of benchmark image datasets--reflecting different abstract visual situations--will be collected to evaluate the recognition system. In addition, the PI will design and run a public competition on automated recognition of visual situations, using the collected datasets. This competition will spur research on this topic, and help researchers working in this area evaluate the success of various methods and gauge the current state of the art on abstract visual recognition. All source code and benchmarking databases developed in this project will be made publicly available via the web.