Collaborative Research: RI: Medium: Bridging the Semantic-Metric Gap via Multinocular Image Integration

人类和其他动物可以毫不费力地和下意识地从视频图像流到他们的眼睛重建周围的3D世界，并成功地将其用于导航，寻找食物，躲避捕食者等计算机视觉3D技术已经迅速发展到从一组摄像机重建世界并在环境中定位这些摄像机。该技术是自动驾驶、机器人导航和无人机飞行中的导航基础;是机器人制造、机器人医疗干预等中的操纵基础;计量测量;娱乐行业建模;以及许多其他应用。因此，3D视觉在能力、效率和鲁棒性方面经历了指数级增长。尽管这种显著的增长来自于利用目前可实现的东西，但存在需要解决的基本缺点，以扩大应用范围并增加现有应用的鲁棒性。首先，来自快速移动的相机的图像（例如，无人机和行人）通常是模糊的并且缺乏特征;室内场景和具有无纹理表面或具有重复纹理的表面的其他场景缺乏特征或具有不可区分的特征;并且存在通常超出当前技术的能力的其他示例。第二，图像感测通常具有高度的冗余，这在当前算法中常常被丢弃，从而丧失了使用冗余中固有的高信息内容的机会。 第三，在当前技术中使用的内部表示（其通常是基于点的）与场景的语义表示（其与对底层曲线（例如，脊）和对象的曲面片（面）。本项目旨在弥补这些不足，实现这些目标需要解决若干技术挑战。首先，该项目确定了数值稳定性的概念，目前混淆退化，应彻底研究和分析的关键多视图几何（MVG）的任务。稳定性要求导致了一类新的技术，将实施并随时提供给社区，以帮助避免故障模式，在广泛的选择MVG问题。第二，开发工具来解决非常大的多项式系统是一种使能技术，不仅可以改变多视图几何问题，还可以改变其他科学领域的广泛问题。第三，这些发展将使一种新的MVG方法的基础上的曲线，曲面，和他们的微分几何的相对姿态估计，绝对姿态估计，和3D重建。这将有助于弥合几何精确的3D点云/网格与对象、对象部件、空间布局、映射等方面的语义有意义的组织之间存在的语义度量差距。多目视觉系统中大量视图信息的有效和可靠集成。该奖项反映了NSF的法定使命，通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Humans and other animals can effortlessly and subconsciously reconstruct the 3D world around them from the video imagery streaming to their eyes, and successfully use it for navigation, food-finding, predator avoidance, etc. Computer vision 3D technology has been evolving rapidly to reconstruct the world from a set of cameras and locate these cameras in the environment. This technology is a basis of navigation as in automated driving, robot navigation, and drone flights; a basis of manipulation as in robotic manufacturing, robotic medical interventions, etc.; measurement in metrology; modeling for the entertainment industry; and a host of other applications. As a result, 3D vision has experienced an exponential growth in capability, efficiency, and robustness. Despite this phenomenal growth arising from exploiting what is currently achievable, fundamental shortcomings exist that need to be addressed to enlarge the scope of application and to increase robustness in existing ones. First, images from rapidly moving cameras (e.g., drones and pedestrians) are often blurry and lack features; indoor scenes and others which have textureless surfaces or surfaces with repeated texture lack features or have indistinguishable features; and there are other examples which are often beyond the capabilities of current technologies. Second, image sensing typically enjoys a high degree of redundancy which is often discarded in current algorithms, thus forfeiting the opportunity to use the high information content inherent in the redundancy. Third, there is often a large gap between the internal representations used in the current technology, which are often point-based, and a semantic representation of the scene, which are more resonant with an understanding of underlying curves (e.g., ridges) and surface patches (faces) of an object. This project aims to remedy these shortcomings.Several technical challenges need to be addressed to achieve these goals. First, this project identifies that the notion of numerical stability, currently confounded with degeneracy, should be thoroughly studied and analyzed for key multiview geometry (MVG) tasks. The stability requirement leads to a new class of techniques which will be implemented and made readily available to the community to help avoid failure modes in a broad selection of MVG problems. Second, the development of tools to solve very large polynomial systems is an enabling technology that will transform not just multiview geometry problems, but also a broad range problem from other scientific areas. Third, these developments will enable a novel MVG approach based on curves, surfaces, and their differential geometry for relative pose estimation, absolute pose estimation, and 3D reconstruction. This will serve to bridge the semantic-metric gap that exists between geometrically accurate 3D point clouds/meshes and semantically meaningful organizations in terms of objects, object parts, spatial layout, mapping, etc. In conjunction, these three streams of research will allow direct, efficient and reliable integration of information across a large number of views in multinocular vision systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.