SFM++: Toward Active, Collaborative, and Semantic Structure-from-Motion (SFM)

SFM：迈向主动、协作和运动语义结构 (SFM)

• 批准号: RGPIN-2014-06686
• 负责人: Tan, Ping
• 金额: $ 3.93万
• 依托单位: Simon Fraser University
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2019
• 资助国家: 加拿大
• 起止时间: 2019-01-01 至 2020-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=690471
• 关键词: SFM++; Toward; Active; Collaborative; Semantic

Structure-from-Motion (SFM) techniques recover 3D information from 2D images. SFM has many applications, including 3D sensing in robotics and 3D content creation in computer games or movies. This proposal studies SFM from a unique perspective. I will take input from multiple independently moving video cameras with wide baselines to address 3D vision challenges. In comparison, previous methods often employ either a single camera or an array of fixed cameras with short baselines. These moving cameras can be handheld camcorders or wireless cameras mounted on different robot platforms. SFM in this novel setting is largely unexplored except in a few of our recent works [3, 4]. As demonstrated in these two works, this new SFM technique can be potentially used to guide a robot team for vision-based autonomous navigation or to provide a flexible handheld camcorder-based 3D reconstruction solution. This study has the potential to make a profound impact. It is the first attempt to develop a collaborative 3D vision system for a team of miniature robots. **Robot vision is possibly the most important application of computer vision. Vision powered autonomous robots will bring revolutionary changes to society and benefit everybody. Due to the difficulties in computer vision, existing robot platforms often rely on bulky laser or ultrasonic scanners to solve the 3D sensing problem. However, these sensors are too heavy or energy inefficient for a miniature platform, e.g. a miniature robot with a payload of less than 50g. Vision is the ideal solution for 3D sensing on these miniature robots. Thus, the robotics and computer vision community devote tremendous efforts to designing 3D vision systems for robots. Almost all of these works focus on a single robot. To tackle difficult 3D vision problems, this proposal studies the collaborative vision sensing of a robot team. By exploiting the collaboration across multiple cameras, my students and I aim to develop robust and efficient computational algorithms for various 3D vision tasks. Specifically, we will address the robot team exploration problem to actively plan their navigation path to enhance 3D sensing. This includes maintaining view overlap between different robots and reducing measurement uncertainty. We will further develop real-time dense 3D reconstruction algorithms from multiple moving cameras, which can recover per-pixel depth information even under complicated dynamic environments. It effectively turns cameras into 3D scanners. We further plan to estimate skeletal human motion from the recovered 3D data by the collaboration of multiple cameras. This proposal will enable a team of miniature robots to navigate autonomously in complicated environments with moving objects. The robot team can sense skeletal human motion which enables natural human robot interaction. We will integrate and demonstrate our results with existing toy or commercial miniature unmanned aerial vehicles (UAVs) such as AR.Drone and Pelican. These algorithms can also be used for applications in computer graphics, such as flexible motion capture with handheld camcorders.**This program is globally unique. The 3D vision of a collaborative robot team is a largely unexplored field. It will clearly extend Canada's lead in this area. Its results can be used in robotics applications for security/surveillance. The research result on 3D skeletal capture from handheld videos can be applied for flexible motion capture in the computer gaming industry, a major employer in Canada. Three PhD students will be trained in this project.

运动恢复结构（SFM）技术从2D图像恢复3D信息。SFM有许多应用，包括机器人中的3D传感和计算机游戏或电影中的3D内容创建。该提案从一个独特的角度研究可持续森林管理。我将从多个独立移动的视频摄像机中获取输入，这些摄像机具有宽基线，以应对3D视觉挑战。相比之下，以前的方法通常采用单个相机或具有短基线的固定相机阵列。这些移动摄像机可以是手持摄像机或安装在不同机器人平台上的无线摄像机。除了我们最近的几部作品外，这种小说背景下的可持续森林管理在很大程度上是未探索的[3，4]。正如这两个作品中所展示的那样，这种新的SFM技术可以潜在地用于引导机器人团队进行基于视觉的自主导航或提供灵活的基于手持摄像机的3D重建解决方案。这项研究有可能产生深远的影响。这是第一次尝试开发一个协作的三维视觉系统的微型机器人团队。** 机器人视觉可能是计算机视觉最重要的应用。视觉驱动的自主机器人将为社会带来革命性的变化，造福每一个人。由于计算机视觉的困难，现有的机器人平台通常依赖于笨重的激光或超声波扫描仪来解决3D传感问题。然而，这些传感器对于微型平台（例如，具有小于50 g的有效载荷的微型机器人）来说太重或能量效率低。Vision是这些微型机器人进行3D传感的理想解决方案。因此，机器人和计算机视觉社区投入了巨大的努力来设计机器人的3D视觉系统。几乎所有这些工作都集中在一个机器人上。为了解决困难的3D视觉问题，该建议研究了机器人团队的协作视觉传感。通过利用多个相机之间的协作，我和我的学生旨在为各种3D视觉任务开发强大而高效的计算算法。具体来说，我们将解决机器人团队探索问题，积极规划他们的导航路径，以增强3D感知。这包括保持不同机器人之间的视图重叠并降低测量不确定性。我们将进一步开发多个移动摄像机的实时密集3D重建算法，即使在复杂的动态环境下也可以恢复每个像素的深度信息。它有效地将相机变成了3D扫描仪。我们还计划通过多个摄像机的协作从恢复的3D数据中估计骨骼人体运动。这一提议将使一组微型机器人能够在有移动物体的复杂环境中自主导航。机器人团队可以感知骨骼人类运动，从而实现自然的人机交互。我们将整合和展示我们的结果与现有的玩具或商业微型无人机（无人机），如AR。无人机和派力肯。这些算法也可以用于计算机图形学中的应用，例如手持摄像机的灵活运动捕捉。该方案在全球独一无二。协作机器人团队的3D视觉在很大程度上是一个未开发的领域。这将明显扩大加拿大在这一领域的领先地位。其结果可用于机器人安全/监控应用。从手持视频的3D骨骼捕捉的研究成果可以应用于灵活的动作捕捉在电脑游戏行业，在加拿大的主要雇主。三名博士生将在该项目中接受培训。