Scene manipulation in the context of uncalibrated images

图像包含了大量丰富而集中的信息，这些信息通常可以被人类立即破译。相比之下，尽管迄今为止取得了显著的进展，但人工视觉系统对图像的理解仍然有限。该研究计划旨在推进当前的视觉系统，使其能够操纵场景的部分，其中后者由一系列二维未校准图像表示。特别是，我们将处理以下三个问题：添加新对象（增强现实），删除现有对象以及在场景中移动/更改对象。在这个建议中，我们考虑了用未知相机拍摄图像的困难情况，并且唯一要使用的数据是像素数组。此外，这里提出的解决方案都是基于视觉的，不需要专门的设备。除了科学进步方面，这项研究的结果可以应用于从远程操作到娱乐等领域。更确切地说，我们将调查以下相关问题。(1)图像中虚拟物体的配准。目前，有一些基于视觉的解决方案，使用校准相机来解决这个问题。然而，当我们拥有的唯一数据由未校准的图像组成时，配准变得复杂。在这种情况下，我们需要检索一些关于观察场景的度量（欧几里得）信息，以实现正确的配准。我们将研究两种可能性来获得这样的度量信息：要么通过使用最新的相机自校准技术，要么通过使用场景/物体几何中的欧几里得约束。(2)跨图像分割和目标跟踪。移除或置换物体需要对目标物体进行适当的分割和识别。后者应该在图像之间进行识别和匹配，以便能够将其移动到不同的位置。(3)更新整个图像序列。一旦通过添加新对象或通过移除/置换现有对象改变了前两个图像，则必须相应地更新序列的其余部分。

Images contain a considerable amount of rich and concentrated information that is usually deciphered immediately by humans. In comparison and despite the remarkable progress made so far, image understanding by artificial vision systems remains limited. This research proposal aims at advancing current vision systems to make them capable of manipulating parts of a scene, where the latter is represented by a sequence of two-dimensional uncalibrated images. In particular, we will deal with the following three problems: adding new objects(augmented reality), removing existing objects and, moving/changing objects within the scene. In this proposal, we consider the difficult case where images are taken with unknown cameras and the only data to be used are the arrays of pixels. Furthermore, the solutions to be proposed here will be all vision-based, with no need for specialized equipment. In addition to the scientific advancement aspect, the results from this research could be applied in areas ranging from teleoperation to entertainment. More precisely, we will investigate the following related problems.(1) Registration of virtual objects in the images. Currently, some vision-based solutions, which use calibrated cameras exist for this problem. However, the registration becomes complex when the only data we have consist of uncalibrated images. In this case, we need to retrieve some metric (Euclidean) information about the observed scene in order to achieve correct registration. We will investigate the two possibilities to get such metric information: either by using recent techniques for camera self-calibration or by using Euclidean constraints from the scene/object geometry.(2) Segmentation and object tracking across images. Removing or displacing objects requires proper segmentation and identification of the target objects. The latter should be identified and matched across the images, in order to be able to move it to a different location.(3) Updating the whole image sequence. Once the first two images have been changed, either by the addition of new objects or by the removal/displacement of existing objects, the rest of the sequence must be updated accordingly.