SFM++: Toward Active, Collaborative, and Semantic Structure-from-Motion (SFM)

运动结构（SFM）技术从二维图像中恢复三维信息。SFM有许多应用，包括机器人中的3D传感和电脑游戏或电影中的3D内容创建。本文从一个独特的视角来研究SFM。我将从多个具有宽基线的独立移动摄像机中获取输入，以解决3D视觉挑战。相比之下，以前的方法通常使用单个摄像机或具有短基线的固定摄像机阵列。这些移动摄像机可以是手持摄像机，也可以是安装在不同机器人平台上的无线摄像机。除了我们最近的一些作品外，这种小说背景下的SFM在很大程度上没有被探索过[3,4]。正如这两项工作所展示的那样，这种新的SFM技术可以潜在地用于指导机器人团队进行基于视觉的自主导航或提供灵活的基于手持摄像机的3D重建解决方案。这项研究有可能产生深远的影响。这是为一组微型机器人开发协作3D视觉系统的首次尝试。

Structure-from-Motion (SFM) techniques recover 3D information from 2D images. SFM has many applications, including 3D sensing in robotics and 3D content creation in computer games or movies. This proposal studies SFM from a unique perspective. I will take input from multiple independently moving video cameras with wide baselines to address 3D vision challenges. In comparison, previous methods often employ either a single camera or an array of fixed cameras with short baselines. These moving cameras can be handheld camcorders or wireless cameras mounted on different robot platforms. SFM in this novel setting is largely unexplored except in a few of our recent works [3, 4]. As demonstrated in these two works, this new SFM technique can be potentially used to guide a robot team for vision-based autonomous navigation or to provide a flexible handheld camcorder-based 3D reconstruction solution. This study has the potential to make a profound impact. It is the first attempt to develop a collaborative 3D vision system for a team of miniature robots. Robot vision is possibly the most important application of computer vision. Vision powered autonomous robots will bring revolutionary changes to society and benefit everybody. Due to the difficulties in computer vision, existing robot platforms often rely on bulky laser or ultrasonic scanners to solve the 3D sensing problem. However, these sensors are too heavy or energy inefficient for a miniature platform, e.g. a miniature robot with a payload of less than 50g. Vision is the ideal solution for 3D sensing on these miniature robots. Thus, the robotics and computer vision community devote tremendous efforts to designing 3D vision systems for robots. Almost all of these works focus on a single robot. To tackle difficult 3D vision problems, this proposal studies the collaborative vision sensing of a robot team. By exploiting the collaboration across multiple cameras, my students and I aim to develop robust and efficient computational algorithms for various 3D vision tasks. Specifically, we will address the robot team exploration problem to actively plan their navigation path to enhance 3D sensing. This includes maintaining view overlap between different robots and reducing measurement uncertainty. We will further develop real-time dense 3D reconstruction algorithms from multiple moving cameras, which can recover per-pixel depth information even under complicated dynamic environments. It effectively turns cameras into 3D scanners. We further plan to estimate skeletal human motion from the recovered 3D data by the collaboration of multiple cameras. This proposal will enable a team of miniature robots to navigate autonomously in complicated environments with moving objects. The robot team can sense skeletal human motion which enables natural human robot interaction. We will integrate and demonstrate our results with existing toy or commercial miniature unmanned aerial vehicles (UAVs) such as AR.Drone and Pelican. These algorithms can also be used for applications in computer graphics, such as flexible motion capture with handheld camcorders. This program is globally unique. The 3D vision of a collaborative robot team is a largely unexplored field. It will clearly extend Canada's lead in this area. Its results can be used in robotics applications for security/surveillance. The research result on 3D skeletal capture from handheld videos can be applied for flexible motion capture in the computer gaming industry, a major employer in Canada. Three PhD students will be trained in this project.