Structural analysis and interactive composition of visual media

该项目代表了12所中国顶尖大学和其他几家英国和美国的受邀合作伙伴的联合工作。Internet和其他大型数据库构成了一个可称为“视觉媒体”的重要资源：图像、视频、3D形状模型等等。互联网文本搜索通常会产生有用的结果。然而，寻找视觉媒体可能要困难得多，例如，具有特定内容的视频，或与某人脑海中图像相似的图像。这部分是由于大多数图像搜索是基于文本输入的，部分是由于分类图像的困难。人类很容易“知道”一幅图像包含什么，但计算机对图像的理解需要许多棘手的任务——将一幅图像分成不同的物体，并分析它们的颜色、形状和许多其他属性。更好的视觉媒体搜索解决方案将支持除了搜索本身之外的许多应用程序，我们还将关注其中之一——在创建新视觉媒体时重用现有的视觉媒体。这个项目有四个主要目标。首先是研究视觉媒体结构分析的新方法。这将包括设计找到重要信息的方法(例如，主要对象是什么？什么是不相关的背景？这个物体是如何由各个部分组成的？)，以及在不同尺度上处理信息的方法（例如，小细节可能与整体形状一样重要）。目的是提出视觉媒体中重要信息的分层描述。第二是在上述层次描述的基础上，找到比较、分类和搜索视觉媒体的有效新方法。我们还将看到如何使用草图作为一种比文本更强大的手段，让用户在搜索时描述他们想要找到的东西。第三个需要考虑的领域是视觉媒体的编辑和重新合成。结构分析将提供更有意义的方法来选择图像的部分，而不仅仅是，例如，用某种颜色选择场景的所有部分。反过来，这将简化编辑视觉媒体的过程。用户将能够对具有相似含义的场景元素应用一致的编辑（例如，用户控制一根手指的弯曲，而计算机对手的其余手指应用类似的弯曲，尽管形状略有不同）。更强大的搜索功能还将允许从视觉媒体数据库或互联网中快速检索元素，以组合成新的场景，或包含在现有图像中，并对不同的灯光进行适当的调整等。当对视频进行处理时，需要进一步考虑以确保结果随着时间的推移是一致的，并且随着时间的推移平滑地变化；视频处理中涉及的大量数据使这成为一个具有挑战性的问题。最后一个工作领域涉及使用机器学习技术来协助实现所有前面的目标。这里的目标是自动学习识别复杂的模式，允许软件根据视觉数据做出智能决策。最终，必须达到一种谨慎的平衡，在这种平衡中，用户牢牢地控制着创作过程，而计算机使用户更容易产生期望的结果。

This project represents joint work between 12 leading Chinese Universities, and several other invited key partners in the UK and US. The Internet, and other large-scale databases, form a significant resource of what may be termed "visual media": images, videos, 3D shape models, and so on. Internet text searches usually produce useful results. However, it can be much more difficult to find visual media, e.g. videos with specific content, or images similar to a picture in one's mind's eye. This is partly due to the fact that most image search is based on text inputs, and partly due to the difficulty of classifying pictures. It is easy for humans to "know" what an image contains, but image understanding by computer requires many tricky tasks - splitting an image into separate objects, and analysing their colour, their shape, and many other attributes. Better solutions to search of visual media would enable many applications in addition to search itself, and we will also look at one of them - the re-use of existing visual media when creating new visual media. This project has four main goals. The first is to investigate new approaches to structural analysis of visual media. This will include devising methods to find salient information (for example, what is the main object? what is irrelevant background? how is this object composed of parts?), and methods which process the information on different scales (small details may be just as important as overall shape, for example). The aim is to come up with hierarchical descriptions of the important information in visual media. The second is to find efficient new approaches to comparing, classifying and searching visual media, based on the above hierarchical descriptions. We will also look at how sketches can be used as a much more powerful means than text of allowing users to describe what they want to find when searching.The third area to be considered is editing and resynthesis of visual media. Structural analysis will provide more meaningful ways to select parts of an image than just, for example, all parts of the scene with a certain colour. In turn, this will simplify the process of editing visual media. Users will be able to apply consistent editing to scene elements with similar meaning (e.g. the user controls bending of one finger, and the computer applies a similar bend to the rest of the fingers of a hand, despite minor shape differences). More powerful search will also allow elements to be rapidly retrieved from visual media databases or the Internet to be combined into new scenes, or to be included within existing images, with suitable adjustment for different lighting, etc. When video is processed, further considerations will be needed to ensure results are consistent over time, and smoothly vary as time progresses; the vast amounts of data involved in video processing make this a challenging problem.The final area of work concerns the use of machine learning techniques to assist with all of the previous goals. The aim here is to automatically learn to recognize complex patterns, permitting software to make intelligent decisions based on visual data. Ultimately, a careful balance must be struck in which the user is firmly in control of the creative process, but the computer makes it easy for the user to produce the desired results.