Scene manipulation in the context of uncalibrated images

图像包含了大量丰富而集中的信息，这些信息通常可以立即被人类破译。相比之下，尽管到目前为止取得了显着的进展，人工视觉系统的图像理解仍然有限。这项研究提案旨在推进当前的视觉系统，使它们能够操纵场景的一部分，后者由一系列二维未校准图像表示。特别是，我们将处理以下三个问题：添加新对象（增强现实），删除现有对象，并在场景中移动/更改对象。在这个建议中，我们考虑了困难的情况下，图像是用未知的相机和唯一要使用的数据是像素阵列。此外，这里提出的解决方案将完全基于视觉，不需要专门的设备。除了科学进步方面，这项研究的结果还可以应用于从远程操作到娱乐等领域。更准确地说，我们将研究以下相关问题。(1)图像中虚拟对象的配准。目前，一些基于视觉的解决方案，使用校准的相机存在这个问题。然而，当我们拥有的唯一数据由未校准的图像组成时，配准变得复杂。在这种情况下，我们需要检索一些度量（欧几里德）信息的观察场景，以实现正确的注册。我们将调查两种可能性，以获得这样的度量信息：无论是通过使用最近的技术相机自校准或通过使用欧几里德约束的场景/对象的几何形状。(2)跨图像的分割和对象跟踪。移除或移动物体需要对目标物体进行适当的分割和识别。应在图像中识别和匹配后者，以便能够将其移动到不同的位置。(3)更新整个图像序列。一旦前两个图像被改变，无论是通过添加新对象还是通过移除/置换现有对象，序列的其余部分必须相应地更新。

Images contain a considerable amount of rich and concentrated information that is usually deciphered immediately by humans. In comparison and despite the remarkable progress made so far, image understanding by artificial vision systems remains limited. This research proposal aims at advancing current vision systems to make them capable of manipulating parts of a scene, where the latter is represented by a sequence of two-dimensional uncalibrated images. In particular, we will deal with the following three problems: adding new objects(augmented reality), removing existing objects and, moving/changing objects within the scene. In this proposal, we consider the difficult case where images are taken with unknown cameras and the only data to be used are the arrays of pixels. Furthermore, the solutions to be proposed here will be all vision-based, with no need for specialized equipment. In addition to the scientific advancement aspect, the results from this research could be applied in areas ranging from teleoperation to entertainment. More precisely, we will investigate the following related problems.(1) Registration of virtual objects in the images. Currently, some vision-based solutions, which use calibrated cameras exist for this problem. However, the registration becomes complex when the only data we have consist of uncalibrated images. In this case, we need to retrieve some metric (Euclidean) information about the observed scene in order to achieve correct registration. We will investigate the two possibilities to get such metric information: either by using recent techniques for camera self-calibration or by using Euclidean constraints from the scene/object geometry.(2) Segmentation and object tracking across images. Removing or displacing objects requires proper segmentation and identification of the target objects. The latter should be identified and matched across the images, in order to be able to move it to a different location.(3) Updating the whole image sequence. Once the first two images have been changed, either by the addition of new objects or by the removal/displacement of existing objects, the rest of the sequence must be updated accordingly.