Collaborative Research: RI: Medium: Bridging the Semantic-Metric Gap via Multinocular Image Integration

人类和其他动物可以毫不费力地、下意识地从流到眼睛的视频图像中重建周围的3D世界，并成功地将其用于导航、寻找食物、躲避捕食者等。计算机视觉3D技术正在迅速发展，从一组摄像机重建世界，并在环境中定位这些摄像机。这项技术是自动驾驶、机器人导航和无人机飞行等导航技术的基础；操作的基础，如机器人制造，机器人医疗干预等；计量学中的测量；娱乐业的模特；还有很多其他的应用。因此，3D视觉在能力、效率和稳健性方面都经历了指数级的增长。尽管利用目前可实现的成果带来了显著的增长，但仍存在需要解决的根本缺陷，以扩大应用范围并增强现有应用的健壮性。首先，来自快速移动的摄像机（例如无人机和行人）的图像通常是模糊的，缺乏特征；室内场景和其他具有无纹理表面或重复纹理表面的场景缺乏特征或具有难以区分的特征；还有一些其他的例子通常超出了当前技术的能力。其次，图像传感通常具有高度的冗余，这在当前算法中经常被丢弃，从而丧失了利用冗余中固有的高信息含量的机会。第三，在当前技术中使用的内部表示（通常是基于点的）和场景的语义表示（与对对象的底层曲线（例如，脊）和表面斑块（面）的理解更有共鸣）之间通常存在很大差距。这个项目旨在弥补这些缺点。要实现这些目标，需要解决若干技术挑战。首先，该项目确定了数值稳定性的概念，目前与简并性混淆，应该彻底研究和分析关键的多视图几何（MVG）任务。稳定性要求导致了一类新的技术，这些技术将被实现并随时提供给社区，以帮助避免在广泛选择的MVG问题中出现故障模式。其次，解决非常大的多项式系统的工具的开发是一种使能技术，它不仅将改变多视图几何问题，而且还将改变其他科学领域的广泛问题。第三，这些发展将使基于曲线、曲面及其微分几何的新型MVG方法成为可能，用于相对姿态估计、绝对姿态估计和3D重建。这将有助于弥合存在于几何上精确的3D点云/网格和在对象、对象部件、空间布局、映射等方面的语义上有意义的组织之间的语义度量差距。结合起来，这三个研究流将允许在多眼视觉系统中直接、有效和可靠地整合大量视图中的信息。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Humans and other animals can effortlessly and subconsciously reconstruct the 3D world around them from the video imagery streaming to their eyes, and successfully use it for navigation, food-finding, predator avoidance, etc. Computer vision 3D technology has been evolving rapidly to reconstruct the world from a set of cameras and locate these cameras in the environment. This technology is a basis of navigation as in automated driving, robot navigation, and drone flights; a basis of manipulation as in robotic manufacturing, robotic medical interventions, etc.; measurement in metrology; modeling for the entertainment industry; and a host of other applications. As a result, 3D vision has experienced an exponential growth in capability, efficiency, and robustness. Despite this phenomenal growth arising from exploiting what is currently achievable, fundamental shortcomings exist that need to be addressed to enlarge the scope of application and to increase robustness in existing ones. First, images from rapidly moving cameras (e.g., drones and pedestrians) are often blurry and lack features; indoor scenes and others which have textureless surfaces or surfaces with repeated texture lack features or have indistinguishable features; and there are other examples which are often beyond the capabilities of current technologies. Second, image sensing typically enjoys a high degree of redundancy which is often discarded in current algorithms, thus forfeiting the opportunity to use the high information content inherent in the redundancy. Third, there is often a large gap between the internal representations used in the current technology, which are often point-based, and a semantic representation of the scene, which are more resonant with an understanding of underlying curves (e.g., ridges) and surface patches (faces) of an object. This project aims to remedy these shortcomings.Several technical challenges need to be addressed to achieve these goals. First, this project identifies that the notion of numerical stability, currently confounded with degeneracy, should be thoroughly studied and analyzed for key multiview geometry (MVG) tasks. The stability requirement leads to a new class of techniques which will be implemented and made readily available to the community to help avoid failure modes in a broad selection of MVG problems. Second, the development of tools to solve very large polynomial systems is an enabling technology that will transform not just multiview geometry problems, but also a broad range problem from other scientific areas. Third, these developments will enable a novel MVG approach based on curves, surfaces, and their differential geometry for relative pose estimation, absolute pose estimation, and 3D reconstruction. This will serve to bridge the semantic-metric gap that exists between geometrically accurate 3D point clouds/meshes and semantically meaningful organizations in terms of objects, object parts, spatial layout, mapping, etc. In conjunction, these three streams of research will allow direct, efficient and reliable integration of information across a large number of views in multinocular vision systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.