Merging Real and Virtual Worlds Considering Depth Information

本课题取得了以下研究成果：1.利用双目立体获取真实场景的三维信息增强现实中最重要的问题之一是解决真实物体和虚拟物体之间的遮挡问题。这个问题需要真实世界场景的深度信息。对如何提高立体匹配的效率进行了研究。我们提高效率的想法是将立体匹配区域限制在虚拟对象合并的区域。2.运动立体获取真实场景的三维信息提出了一种新的因子分解方法，用于从运动摄像机获取的一系列图像中估计真实场景的三维结构。另一种尝试是预测动态图像中的表观运动，提高了立体匹配的效率。3.沉浸式模型生成虚拟对象我们扩展了沉浸式实体建模工具VLEGO，使其能够以自然简单的方式生成虚拟对象。还开发了一种用于对虚拟对象施加几何约束的技术。这使得创建关节对象和生成动画成为可能。4.将虚拟对象合并到具有正确遮挡的真实世界场景图像中通过跟踪捕获图像中的标记来实现真实世界和虚拟世界之间的几何配准。这可以被称为基于视觉的注册。结合上述研究成果，我们提出了一种有效的立体图像合成方法。5.实、虚拟融合原型系统的开发我们开发了一个立体视频透视式头盔显示器原型系统，能够实时融合真实物体和虚拟物体，并具有正确的遮挡。实验证明了原型系统的可行性。融合真实和虚拟的显示图像每100ms更新一次。

We have obtained the following research results in this project.1.Acquiring 3D Information of a Real Scene by Binocular StereoOne of the most important problems in augmented reality is to resolve occlusions between real and virtual objects. The problem requires depth information of a real world scene. We have studied on improving the efficiency of stereo matching. Our idea for improving the efficiency is to limit stereo matching areas to those on which virtual objects are merged. This made it possible to estimate depth of real world in real time.2.Acquiring 3D Information of a Real Scene by Motion StereoA new factorization method was developed for estimating 3D structure of a real scene from a sequence of images obtained by a moving camera. Another attempt is to predict apparent motion in dynamic images, which improves the efficiency of stereo matching.3.Generation of Virtual Objects by Immersive ModelerWe have extended the immersive solid modeler VLEGO for generating virtual objects in natural and simple way. Also developed is a technique for imposing geometric constraints on virtual objects. This made it possible to create articulated objects and to generate animations.4.Merging Virtual Objects into a Real World Scene Image with Correct OcclusionGeometric registration between real and virtual worlds was achieved by tracking markers in captured images. This may be called a vision-based registration. Integrating this result with those in above items, we have developed an efficient method of stereo image composition.5.Development of Prototype System for Merging Real and VirtualWe have developed a prototype of stereoscopic video see-through HMD which can merge real and virtual objects with correct occlusion in real time. Experiments have shown the feasibility of the prototype systems. The displayed image of merging real and virtual is updated every 100ms.