SFM++: Toward Active, Collaborative, and Semantic Structure-from-Motion (SFM)

运动结构（SFM）技术从二维图像中恢复三维信息。SFM有许多应用，包括机器人中的3D传感和电脑游戏或电影中的3D内容创建。本文从一个独特的视角来研究SFM。我将从多个具有宽基线的独立移动摄像机中获取输入，以解决3D视觉挑战。相比之下，以前的方法通常使用单个摄像机或具有短基线的固定摄像机阵列。这些移动摄像机可以是手持摄像机，也可以是安装在不同机器人平台上的无线摄像机。除了我们最近的一些作品外，这种小说背景下的SFM在很大程度上没有被探索过[3,4]。正如这两项工作所展示的那样，这种新的SFM技术可以潜在地用于指导机器人团队进行基于视觉的自主导航或提供灵活的基于手持摄像机的3D重建解决方案。这项研究有可能产生深远的影响。这是为一组微型机器人开发协作3D视觉系统的首次尝试。**机器人视觉可能是计算机视觉最重要的应用。视觉驱动的自主机器人将给社会带来革命性的变化，造福于每个人。由于计算机视觉的困难，现有的机器人平台往往依靠体积庞大的激光或超声波扫描仪来解决三维传感问题。然而，对于微型平台来说，这些传感器太重或能源效率低下，例如有效载荷小于50克的微型机器人。视觉是这些微型机器人3D传感的理想解决方案。因此，机器人技术和计算机视觉界为设计机器人的三维视觉系统付出了巨大的努力。几乎所有这些工作都集中在一个机器人上。为了解决复杂的三维视觉问题，本文研究了机器人团队的协同视觉感知。通过利用多个相机之间的协作，我和我的学生的目标是为各种3D视觉任务开发强大而高效的计算算法。具体而言，我们将解决机器人团队探索问题，积极规划其导航路径，增强3D感知。这包括保持不同机器人之间的视图重叠和减少测量不确定性。我们将进一步开发基于多个移动摄像机的实时密集三维重建算法，即使在复杂的动态环境下也能恢复每像素深度信息。它有效地将相机变成了3D扫描仪。我们进一步计划通过多个摄像机的协作，从恢复的3D数据中估计骨骼人体运动。这一提议将使一组微型机器人能够在有移动物体的复杂环境中自主导航。机器人团队可以感知人体骨骼的运动，从而实现自然的人机交互。我们将整合并展示现有的玩具或商业微型无人机（uav），如AR.Drone和Pelican。这些算法也可以用于计算机图形学的应用，例如手持摄像机的灵活动作捕捉。**该计划是全球唯一的。协作机器人团队的3D视觉在很大程度上是一个尚未开发的领域。这显然将扩大加拿大在这一领域的领先地位。其结果可用于安全/监视机器人应用。从手持视频中获取3D骨骼的研究结果可以应用于加拿大主要雇主电脑游戏行业的灵活动作捕捉。本项目将培养3名博士生。

Structure-from-Motion (SFM) techniques recover 3D information from 2D images. SFM has many applications, including 3D sensing in robotics and 3D content creation in computer games or movies. This proposal studies SFM from a unique perspective. I will take input from multiple independently moving video cameras with wide baselines to address 3D vision challenges. In comparison, previous methods often employ either a single camera or an array of fixed cameras with short baselines. These moving cameras can be handheld camcorders or wireless cameras mounted on different robot platforms. SFM in this novel setting is largely unexplored except in a few of our recent works [3, 4]. As demonstrated in these two works, this new SFM technique can be potentially used to guide a robot team for vision-based autonomous navigation or to provide a flexible handheld camcorder-based 3D reconstruction solution. This study has the potential to make a profound impact. It is the first attempt to develop a collaborative 3D vision system for a team of miniature robots. **Robot vision is possibly the most important application of computer vision. Vision powered autonomous robots will bring revolutionary changes to society and benefit everybody. Due to the difficulties in computer vision, existing robot platforms often rely on bulky laser or ultrasonic scanners to solve the 3D sensing problem. However, these sensors are too heavy or energy inefficient for a miniature platform, e.g. a miniature robot with a payload of less than 50g. Vision is the ideal solution for 3D sensing on these miniature robots. Thus, the robotics and computer vision community devote tremendous efforts to designing 3D vision systems for robots. Almost all of these works focus on a single robot. To tackle difficult 3D vision problems, this proposal studies the collaborative vision sensing of a robot team. By exploiting the collaboration across multiple cameras, my students and I aim to develop robust and efficient computational algorithms for various 3D vision tasks. Specifically, we will address the robot team exploration problem to actively plan their navigation path to enhance 3D sensing. This includes maintaining view overlap between different robots and reducing measurement uncertainty. We will further develop real-time dense 3D reconstruction algorithms from multiple moving cameras, which can recover per-pixel depth information even under complicated dynamic environments. It effectively turns cameras into 3D scanners. We further plan to estimate skeletal human motion from the recovered 3D data by the collaboration of multiple cameras. This proposal will enable a team of miniature robots to navigate autonomously in complicated environments with moving objects. The robot team can sense skeletal human motion which enables natural human robot interaction. We will integrate and demonstrate our results with existing toy or commercial miniature unmanned aerial vehicles (UAVs) such as AR.Drone and Pelican. These algorithms can also be used for applications in computer graphics, such as flexible motion capture with handheld camcorders.**This program is globally unique. The 3D vision of a collaborative robot team is a largely unexplored field. It will clearly extend Canada's lead in this area. Its results can be used in robotics applications for security/surveillance. The research result on 3D skeletal capture from handheld videos can be applied for flexible motion capture in the computer gaming industry, a major employer in Canada. Three PhD students will be trained in this project.