Structural analysis and interactive composition of visual media

该项目代表着12所中国顶尖大学与英国和美国其他几个受邀的关键合作伙伴共同努力的成果。互联网和其他大型数据库形成了一个重要的“视觉媒体”资源：图像、视频、3D形状模型等。互联网文本搜索通常会产生有用的结果。然而，找到视觉媒体可能要困难得多，例如具有特定内容的视频，或者与人们脑海中的眼睛中的图片相似的图像。这部分是因为大多数图像搜索是基于文本输入的，部分是因为对图片进行分类的难度。人类很容易“知道”图像包含什么，但用计算机理解图像需要许多棘手的任务--将图像分割成不同的对象，并分析它们的颜色、形状和许多其他属性。更好的视觉媒体搜索解决方案将支持除搜索本身之外的许多应用程序，我们还将研究其中之一-在创建新的视觉媒体时重新使用现有的视觉媒体。这个项目有四个主要目标。第一个是研究视觉媒体结构分析的新方法。这将包括设计寻找显著信息的方法(例如，主要对象是什么？什么是不相关的背景？这个对象是如何由部分组成的？)，以及在不同尺度上处理信息的方法(例如，小细节可能与整体形状一样重要)。其目的是对视觉媒体中的重要信息进行分层描述。二是在上述层次描述的基础上，寻找比较、分类和搜索视觉媒体的有效新方法。我们还将研究如何使用草图作为一种比文本更强大的手段，允许用户描述他们在搜索时想要找到的内容。第三个要考虑的领域是视觉媒体的编辑和重新合成。结构分析将提供更有意义的方法来选择图像的部分，而不仅仅是选择具有某种颜色的场景的所有部分。反过来，这将简化视觉媒体的编辑过程。用户将能够对具有相似含义的场景元素应用一致的编辑(例如，用户控制一个手指的弯曲，而计算机将类似的弯曲应用于手的其余手指，尽管形状差异很小)。更强大的搜索还将允许从视觉媒体数据库或互联网中快速检索元素，以组合成新的场景，或包括在现有图像中，并针对不同的照明等进行适当调整。在处理视频时，将需要进一步考虑以确保结果随时间的变化是一致的，并随时间的进展而平滑变化；视频处理涉及的海量数据使这成为一个具有挑战性的问题。最后的工作领域涉及使用机器学习技术来帮助实现所有先前的目标。这里的目标是自动学习识别复杂的模式，使软件能够基于视觉数据做出智能决策。归根结底，必须在用户牢牢控制创作过程的情况下取得谨慎的平衡，但计算机使用户很容易产生想要的结果。

This project represents joint work between 12 leading Chinese Universities, and several other invited key partners in the UK and US. The Internet, and other large-scale databases, form a significant resource of what may be termed "visual media": images, videos, 3D shape models, and so on. Internet text searches usually produce useful results. However, it can be much more difficult to find visual media, e.g. videos with specific content, or images similar to a picture in one's mind's eye. This is partly due to the fact that most image search is based on text inputs, and partly due to the difficulty of classifying pictures. It is easy for humans to "know" what an image contains, but image understanding by computer requires many tricky tasks - splitting an image into separate objects, and analysing their colour, their shape, and many other attributes. Better solutions to search of visual media would enable many applications in addition to search itself, and we will also look at one of them - the re-use of existing visual media when creating new visual media. This project has four main goals. The first is to investigate new approaches to structural analysis of visual media. This will include devising methods to find salient information (for example, what is the main object? what is irrelevant background? how is this object composed of parts?), and methods which process the information on different scales (small details may be just as important as overall shape, for example). The aim is to come up with hierarchical descriptions of the important information in visual media. The second is to find efficient new approaches to comparing, classifying and searching visual media, based on the above hierarchical descriptions. We will also look at how sketches can be used as a much more powerful means than text of allowing users to describe what they want to find when searching.The third area to be considered is editing and resynthesis of visual media. Structural analysis will provide more meaningful ways to select parts of an image than just, for example, all parts of the scene with a certain colour. In turn, this will simplify the process of editing visual media. Users will be able to apply consistent editing to scene elements with similar meaning (e.g. the user controls bending of one finger, and the computer applies a similar bend to the rest of the fingers of a hand, despite minor shape differences). More powerful search will also allow elements to be rapidly retrieved from visual media databases or the Internet to be combined into new scenes, or to be included within existing images, with suitable adjustment for different lighting, etc. When video is processed, further considerations will be needed to ensure results are consistent over time, and smoothly vary as time progresses; the vast amounts of data involved in video processing make this a challenging problem.The final area of work concerns the use of machine learning techniques to assist with all of the previous goals. The aim here is to automatically learn to recognize complex patterns, permitting software to make intelligent decisions based on visual data. Ultimately, a careful balance must be struck in which the user is firmly in control of the creative process, but the computer makes it easy for the user to produce the desired results.