Collaborative Research: RI: Medium: Bridging the Semantic-Metric Gap via Multinocular Image Integration

人类和其他动物可以毫不费力地、下意识地从流入他们眼睛的视频图像中重建他们周围的3D世界，并成功地将其用于导航、寻找食物、躲避捕食者等。计算机视觉3D技术已经迅速发展，从一组摄像头重建世界，并在环境中定位这些摄像头。这项技术是自动驾驶、机器人导航和无人机飞行中导航的基础；机器人制造、机器人医疗干预等中的操纵基础；计量学中的测量；娱乐业的建模；以及许多其他应用。因此，3D视觉在能力、效率和健壮性方面都经历了指数级的增长。尽管利用目前可以实现的东西取得了惊人的增长，但仍然存在一些根本性的缺陷，需要加以解决，以扩大应用范围并增强现有应用的稳健性。首先，来自快速移动的摄像机(例如无人机和行人)的图像往往模糊且缺乏特征；室内场景和其他具有无纹理表面或具有重复纹理的表面的图像缺乏特征或具有难以区分的特征；还有其他例子往往超出当前技术的能力范围。其次，图像传感通常享有在当前算法中经常被丢弃的高度冗余，从而丧失了使用冗余中固有的高信息量的机会。第三，在当前技术中使用的通常是基于点的内部表示和场景的语义表示之间往往有很大的差距，其中场景的语义表示与对对象的底层曲线(例如，脊)和表面片(面)的理解更有共鸣。这个项目旨在弥补这些缺陷。为了实现这些目标，需要解决几个技术挑战。首先，该项目确定，对于关键的多视图几何(MVG)任务，目前与退化混淆的数值稳定性的概念应该得到彻底的研究和分析。稳定性要求导致了一类新的技术，这些技术将被实施并随时可供社区使用，以帮助避免在广泛的MVG问题中出现故障模式。其次，开发工具来解决非常大的多项式系统是一项使能技术，它不仅将转变多视点几何问题，而且还将转变来自其他科学领域的广泛问题。第三，这些发展将使基于曲线、曲面及其微分几何的MVG方法能够用于相对位姿估计、绝对位姿估计和3D重建。这将有助于弥合几何上精确的3D点云/网格和语义上有意义的组织在对象、对象部分、空间布局、地图等方面存在的语义-度量差距。这三个研究流结合在一起，将允许在多眼视觉系统中的大量视图中直接、高效和可靠地整合信息。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Humans and other animals can effortlessly and subconsciously reconstruct the 3D world around them from the video imagery streaming to their eyes, and successfully use it for navigation, food-finding, predator avoidance, etc. Computer vision 3D technology has been evolving rapidly to reconstruct the world from a set of cameras and locate these cameras in the environment. This technology is a basis of navigation as in automated driving, robot navigation, and drone flights; a basis of manipulation as in robotic manufacturing, robotic medical interventions, etc.; measurement in metrology; modeling for the entertainment industry; and a host of other applications. As a result, 3D vision has experienced an exponential growth in capability, efficiency, and robustness. Despite this phenomenal growth arising from exploiting what is currently achievable, fundamental shortcomings exist that need to be addressed to enlarge the scope of application and to increase robustness in existing ones. First, images from rapidly moving cameras (e.g., drones and pedestrians) are often blurry and lack features; indoor scenes and others which have textureless surfaces or surfaces with repeated texture lack features or have indistinguishable features; and there are other examples which are often beyond the capabilities of current technologies. Second, image sensing typically enjoys a high degree of redundancy which is often discarded in current algorithms, thus forfeiting the opportunity to use the high information content inherent in the redundancy. Third, there is often a large gap between the internal representations used in the current technology, which are often point-based, and a semantic representation of the scene, which are more resonant with an understanding of underlying curves (e.g., ridges) and surface patches (faces) of an object. This project aims to remedy these shortcomings.Several technical challenges need to be addressed to achieve these goals. First, this project identifies that the notion of numerical stability, currently confounded with degeneracy, should be thoroughly studied and analyzed for key multiview geometry (MVG) tasks. The stability requirement leads to a new class of techniques which will be implemented and made readily available to the community to help avoid failure modes in a broad selection of MVG problems. Second, the development of tools to solve very large polynomial systems is an enabling technology that will transform not just multiview geometry problems, but also a broad range problem from other scientific areas. Third, these developments will enable a novel MVG approach based on curves, surfaces, and their differential geometry for relative pose estimation, absolute pose estimation, and 3D reconstruction. This will serve to bridge the semantic-metric gap that exists between geometrically accurate 3D point clouds/meshes and semantically meaningful organizations in terms of objects, object parts, spatial layout, mapping, etc. In conjunction, these three streams of research will allow direct, efficient and reliable integration of information across a large number of views in multinocular vision systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.