RI: Small: Collaborative Research: A Scalable Architecture for Image Interpretation

对视觉图像意义的无缝理解是人类认知的一个关键属性，远远超出了当前计算机视觉程序的能力。这个项目的目的是建立一个计算系统，通过将较高级别的概念与较低级别的视觉感知相结合来捕获人类视觉的动态和交互方面。如果成功，这个系统将能够以一种与场景的复杂性很好地缩放的方式来解释视觉场景。当前的计算机视觉系统通常依赖相对低级的视觉信息(例如，颜色、纹理、形状)来分类对象或确定场景的整体类别。这种分类通常是以自下而上的方式完成的，其中视觉系统从场景的所有部分提取较低级别的特征，并且随后分析所提取的特征以确定场景的哪些部分包含感兴趣的对象以及应该如何对这些对象进行分类。这样的系统缺乏扩展到大量视觉类别以及识别涉及对象类别之间的空间和抽象关系的更复杂的视觉概念的能力。人类的视觉感知是一个有反馈的时间过程，在这个过程中，较低级别的视觉特征用于激活较高级别的概念(或知识)。这些活跃的概念反过来又引导人们对较低级别的视觉特征的感知和关注。此外，激活的概念可以将激活扩展到语义相关的概念(例如，“车轮”可能激活“汽车”或“自行车”；“自行车”可能激活“道路”或“骑手”)。通过这种方式，在较低和较高水平的视觉之间有一种持续的相互作用，这允许观众聚焦并连接复杂场景的重要方面，以便感知其含义，而不必同等关注场景的每一个细节。这里提出的系统将对人类视觉感知的这些方面进行建模。拟议的系统名为Petacat，将整合并建立在两个现有项目的基础上：最初由Riesenuber和Poggio开发的对象识别的HMAX模型，以及由Hofstadter和Mitchell开发的模仿高级感知和类比制作的模型。HMAX通过从图像中提取日益复杂的纹理和形状特征的前馈网络，对哺乳动物视觉皮质的“什么”路径进行建模。(洛斯阿拉莫斯的合成视觉小组已经将HMAX重新实现为“Petascale人工神经网络”或PANN，以便在大量神经元上进行高性能计算。)Copycat实现了高层概念和底层知觉之间的相互作用过程，并已被用来模拟几个非视觉领域的注意焦点、概念滑移和类推。该项目将结合HMAX/PAN的特征提取能力和Copycat的高级交互感知能力来构建Petacat体系结构。Petacat的图像解释能力将根据相关的语义视觉识别任务家族进行评估(例如，以灵活的、类似人类的方式识别“遛狗”的实例)。该项目的评价部分将涉及为语义图像理解系统建立基准的图像数据库。Petacat的源代码和基准数据库将通过网络公开提供。

Seamless understanding of the meaning of visual images is a key property of human cognition that is far beyond the abilities of current computer vision programs. The purpose of this project is to build a computational system that captures the dynamical and interactive aspects of human vision by integrating higher-level concepts with lower-level visual perception. If successful, this system will be able to interpret visual scenes in a way that scales well with the complexity of the scene. Current computer vision systems typically rely on relatively low-level visual information (e.g., color, texture, shape) to classify objects or determine the overall category of a scene. Such categorization is typically done in a "bottom-up" fashion, in which the vision system extracts lower-level features from all parts of the scene, and subsequently analyzes the extracted features to determine which parts of the scene contain objects of interest and how those objects should be categorized. Such systems lack the abilities to scale to large numbers of visual categories and to identify more complex visual concepts that involve spatial and abstract relationships among object categories. Visual perception by humans is known to be a temporal process with feedback, in which lower-level visual features serve to activate higher-level concepts (or knowledge). These active concepts, in turn, guide the perception of and attention given to lower-level visual features. Moreover, activated concepts can spread activation to semantically related concepts (e.g., "wheels" might activate "car" or "bicycle"; "bicycle" might activate "road" or "rider"). In this way there is a continual interaction between the lower and higher levels of vision, which allows the viewer to focus on and connect important aspects of a complex scene in order to perceive its meaning, without having to pay equal attention to every detail of the scene. The system proposed here will model these aspects of human visual perception. The proposed system, called Petacat, will integrate and build on two existing projects: the HMAX model of object recognition originally developed by Riesenhuber and Poggio, and the Copycat model of high-level perception and analogy-making, developed by Hofstadter and Mitchell. HMAX models the "what" pathway of mammalian visual cortex via a feed-forward network that extracts increasingly complex textural and shape features from an image. (HMAX has been reimplemented, as the "Petascale Artificial Neural Network" or PANN, by the Synthetic Vision Group at Los Alamos to allow for high-performance computing on large numbers of neurons.) Copycat implements a process of interaction between high-level concepts and lower-level perception, and has been used to model focus of attention, conceptual slippage, and analogy-making in several non-visual domains. This project will marry the feature extraction abilities of HMAX/PANN with the higher-level interactive perceptual abilities of Copycat to build the Petacat architecture. The image interpretation abilities of Petacat will be evaluated on families of related semantic visual recognition tasks (e.g., recognizing, in a flexible, human-like way, instances of "walking a dog"). The evaluation part of the project will involve the creation of image databases for benchmarking semantic image-understanding systems. The Petacat source code and benchmarking databases will be made publically available via the web.