Perceptual content description language and interface for supporting visual semantic computing

视觉数据的快速增长吸引了认知科学、计算机视觉、信息检索、数据库和机器学习领域的研究兴趣，以开发处理大型图像和视频数据库以及基于其语义内容的视觉数据流的新方法。视觉语义计算（VSC）领域汇集了有关将人类意图（通常是模糊表述）与图像和视频联系起来的学科。各种视觉信息系统的用户可能以不同的方式感知数据，但一个共同的需求是能够根据自己对数据的感知来处理和查询数据。设计能够很好地理解图像以实现有效操作的系统，例如大型数据库的搜索，仍然是一个非常具有挑战性的问题。例如，图像检索系统不仅仅是图像相似度引擎。除了图像匹配，它还应该解决索引问题，以实现快速搜索；考虑先验信息，这可以用来衡量一些图像比其他图像更强；探索用户在检索循环中的存在。尽管近年来一直在努力，但最大的挑战仍然是弥合人类对视觉数据的感知与计算机所看到的特征之间的语义差距。本研究的目标是以对称的方式解决这一挑战，强调基于感知的内容表示和操作。1)探讨人类视觉认知研究在视觉注意、视觉模块、视觉感知和视觉记忆等方面的关键理论和假设。2)以连贯和协作的方式复制一些关键功能，以提取感知特征，特别是基于感知分区和分组。3)在1)和2)的基础上，构建了感知内容描述语言（PCDL）框架。4)在视觉语义分类（VST）的构建和VST与PCDL之间的映射中，建立了基于感知符号的描述与先验语义词相关联的系统方法。这种新方法将促进基于内容的图像/视频检索、索引、注释、操作和识别的VSC应用程序的开发。

The rapid growth of visual data has attracted a significant research interest in cognition science, computer vision, information retrieval, databases and machine learning to develop new methods for handling large image and video databases and visual data streams based on its semantic content. The field of Visual Semantic Computing (VSC) brings together those disciplines concerned with connecting human intention (often vaguely-formulated) with images and video. The users of various visual information systems may perceive the data in different ways, but a common demand is the ability to process and query the data based on their own perception of the data. Designing systems that understand images well enough to enable effective operations, such as search of large databases remains a very challenging problem. E.g., an image retrieval system is more than an image similarity engine. In addition to image matching, it should address the problems of indexing to enable fast searches; accounting for prior information, which can be used to weigh some images more strongly than others; and exploring the user's presence in the retrieval loop. Despite the sustained efforts in recent years, the paramount challenge still remains on bridging the semantic gap between human's perception on visual data and the features seen by the computer. The goal of this research is to tackle this challenge in a symmetrical way emphasizing perception based content representation and manipulation. 1) Investigate the key theories and hypotheses from human vision and cognition research in visual attention, modularity, perception, and memory. 2) Duplicate some of the key functionalities in coherent and collaborative way for extracting perceptual features, in particular Perceptual Partitioning and Grouping based. 3) Develop a frame work of Perceptual Content Description Language (PCDL) based on 1) and 2). 4) Develop a systematic approach for constructing Visual Semantic Taxonomy (VST) and mapping between VST and PCDL in that the perceptual iconic based descriptions are linked to semantic words in prior. This new approach will facilitate VSC application development for content based image/video retrieval, indexing, annotation, manipulation and recognition.