Scene manipulation in the context of uncalibrated images

图像包含相当数量的丰富而集中的信息，通常人类可以立即破译这些信息。相比之下，尽管到目前为止已经取得了显著的进步，但人工视觉系统对图像的理解仍然有限。这一研究方案旨在推进现有的视觉系统，使其能够操作场景的部分，其中场景的部分由一系列二维未标定图像表示。特别是，我们将处理以下三个问题：添加新对象(增强现实)、移除现有对象以及在场景中移动/更改对象。在这个方案中，我们考虑了这样一种困难的情况，即用未知的摄像机拍摄图像，并且唯一要使用的数据是像素阵列。此外，这里提出的解决方案将全部基于愿景，不需要专门的设备。除了科学进步方面，这项研究的结果还可以应用于从遥操作到娱乐的各个领域。更确切地说，我们将研究以下相关问题。(1)图像中虚拟物体的配准。目前，存在一些基于视觉的解决方案，它们使用标定的摄像机来解决这一问题。然而，当我们拥有的唯一数据包括未校准的图像时，配准变得复杂。在这种情况下，我们需要检索有关观察场景的一些度量(欧几里得)信息，以便实现正确的配准。我们将研究获取此类度量信息的两种可能性：使用最新的摄像机自标定技术或使用场景/对象几何的欧几里德约束。(2)跨图像的分割和对象跟踪。移除或移位对象需要对目标对象进行适当的分割和识别。后者应该跨图像进行识别和匹配，以便能够将其移动到不同的位置。(3)更新整个图像序列。一旦前两个图像通过添加新对象或通过移除/移动现有对象而改变，则序列的其余部分必须相应地更新。

Images contain a considerable amount of rich and concentrated information that is usually deciphered immediately by humans. In comparison and despite the remarkable progress made so far, image understanding by artificial vision systems remains limited. This research proposal aims at advancing current vision systems to make them capable of manipulating parts of a scene, where the latter is represented by a sequence of two-dimensional uncalibrated images. In particular, we will deal with the following three problems: adding new objects(augmented reality), removing existing objects and, moving/changing objects within the scene. In this proposal, we consider the difficult case where images are taken with unknown cameras and the only data to be used are the arrays of pixels. Furthermore, the solutions to be proposed here will be all vision-based, with no need for specialized equipment. In addition to the scientific advancement aspect, the results from this research could be applied in areas ranging from teleoperation to entertainment. More precisely, we will investigate the following related problems.(1) Registration of virtual objects in the images. Currently, some vision-based solutions, which use calibrated cameras exist for this problem. However, the registration becomes complex when the only data we have consist of uncalibrated images. In this case, we need to retrieve some metric (Euclidean) information about the observed scene in order to achieve correct registration. We will investigate the two possibilities to get such metric information: either by using recent techniques for camera self-calibration or by using Euclidean constraints from the scene/object geometry.(2) Segmentation and object tracking across images. Removing or displacing objects requires proper segmentation and identification of the target objects. The latter should be identified and matched across the images, in order to be able to move it to a different location.(3) Updating the whole image sequence. Once the first two images have been changed, either by the addition of new objects or by the removal/displacement of existing objects, the rest of the sequence must be updated accordingly.