Large-vocabulary Semantic Image Processing: Theory and Algorithms

经典的图像处理大多忽略了语义图像表示，而倾向于基于低层信号属性(频率分解、均方误差等)的更易于数学处理的表示。这与图像处理问题的生物解决方案不同，后者在很大程度上依赖于对场景内容的理解。例如，面部区域的处理通常比背景中的灌木丛更仔细。不能将图像处理调整到图像内容的语义相关性经常导致将诸如带宽、错误保护或观看时间之类的资源分配到感知上无关的图像区域。语义图像处理系统部署的主要障碍之一是难以训练内容？]理解具有大规模词汇表的系统。这在很大程度上是因为经典的词汇学习方法需要大量的训练数据和密集的人工监督。本研究的目的是为语义图像处理系统从形式标注的数据中学习大规模词汇而不需要额外的人工监督奠定基础。它建立在语义图像标注的最新进展的基础上，这使得从嘈杂的训练数据中学习词汇成为可能，比如网络上大量(和廉价的)可用的训练数据。这项研究既研究了词汇学习的理论问题，也研究了图像处理算法的设计，该算法根据被处理图像的内容来调整他们的行为。语义图像处理可以在图像压缩、增强、加密、去噪或分割等领域带来变革性的进步，这些领域对医疗成像、图像搜索和检索或安全和监视等各种应用都很感兴趣。

Classical image processing has mostly disregarded semantic image representations, in favor of more mathematically tractable representations based on low?]level signal properties (frequency decompositions, mean squared error, etc.). This is unlike biological solutions to image processingproblems, which rely extensively on understanding of scene content. For example, regions of faces are usually processed more carefully than the bushes in the background. The inability to tune image processing to the semantic relevance of image content frequently leads to the sub?]optimal allocation ofresources, such as bandwidth, error protection, or viewing time, to image areas that are perceptually irrelevant. One of the main obstacles to the deployment of semantic image processing systems has been the difficulty of training content?]understanding systems with large scale vocabularies. This is, in great part, due to the requirement for large amounts of training data and intensive human supervision associated with the classical methods for vocabulary learning. This research aims to establish a foundation for semantic image processing systems that can learn large scale vocabularies frominformally annotated data and no additional human supervision. It builds on recent advances in semantic image labeling, which have made it possible to learn vocabularies from noisy training data, such as that massively (and inexpensively) available on the web. The research studies both theoreticalissues in vocabulary learning, and the design of image processing algorithms that tune their behavior according to the content of the images being processed. Semantic image processing could lead to transformative advances in areas such as image compression, enhancement, encryption, de?]noising, orsegmentation, among others, which are of interest for applications as diverse as medical imaging, image search and retrieval, or security and surveillance.