Perceptual content description language and interface for supporting visual semantic computing

视觉数据的快速增长吸引了认知科学、计算机视觉、信息检索、数据库和机器学习等领域的研究兴趣，以开发基于其语义内容的新方法来处理大型图像和视频数据库以及视觉数据流。视觉语义计算(VSC)领域汇集了与人类意图(通常是含糊其辞的表述)与图像和视频相关的学科。各种视觉信息系统的用户可能以不同的方式感知数据，但共同的需求是基于他们自己对数据的感知来处理和查询数据的能力。设计足够好地理解图像以实现有效操作的系统，例如搜索大型数据库，仍然是一个非常具有挑战性的问题。例如，图像检索系统不仅仅是图像相似性引擎。除了图像匹配，它还应该解决索引以实现快速搜索的问题；考虑可以用来对某些图像进行比其他图像更强权重的先验信息；以及探索用户在检索循环中的存在。尽管近年来做出了持续的努力，但最大的挑战仍然是弥合人类对视觉数据的感知和计算机所看到的特征之间的语义鸿沟。本研究的目标是以一种对称的方式应对这一挑战，强调基于感知的内容表示和操纵。1)研究人类视觉和认知研究在视觉注意、模块化、知觉和记忆等方面的关键理论和假设。2)以连贯和协作的方式复制一些关键功能，以提取感知特征，特别是基于感知的划分和分组。3)在1)和2)的基础上提出了感知内容描述语言(PCDL)的框架。4)开发了一种系统的方法来构建视觉语义分类(VST)以及VST和PCDL之间的映射，因为基于知觉图标的描述与先前的语义词相关联。这一新方法将促进基于内容的图像/视频检索、索引、注释、操作和识别的VSC应用程序开发。

The rapid growth of visual data has attracted a significant research interest in cognition science, computer vision, information retrieval, databases and machine learning to develop new methods for handling large image and video databases and visual data streams based on its semantic content. The field of Visual Semantic Computing (VSC) brings together those disciplines concerned with connecting human intention (often vaguely-formulated) with images and video. The users of various visual information systems may perceive the data in different ways, but a common demand is the ability to process and query the data based on their own perception of the data. Designing systems that understand images well enough to enable effective operations, such as search of large databases remains a very challenging problem. E.g., an image retrieval system is more than an image similarity engine. In addition to image matching, it should address the problems of indexing to enable fast searches; accounting for prior information, which can be used to weigh some images more strongly than others; and exploring the user's presence in the retrieval loop. Despite the sustained efforts in recent years, the paramount challenge still remains on bridging the semantic gap between human's perception on visual data and the features seen by the computer. The goal of this research is to tackle this challenge in a symmetrical way emphasizing perception based content representation and manipulation. 1) Investigate the key theories and hypotheses from human vision and cognition research in visual attention, modularity, perception, and memory. 2) Duplicate some of the key functionalities in coherent and collaborative way for extracting perceptual features, in particular Perceptual Partitioning and Grouping based. 3) Develop a frame work of Perceptual Content Description Language (PCDL) based on 1) and 2). 4) Develop a systematic approach for constructing Visual Semantic Taxonomy (VST) and mapping between VST and PCDL in that the perceptual iconic based descriptions are linked to semantic words in prior. This new approach will facilitate VSC application development for content based image/video retrieval, indexing, annotation, manipulation and recognition.