RI: Small: A Differential Geometry Paradigm for Constructing a Semantic Mid-Level Representation for Multinocular Pose Estimation and Reconstruction

3D视觉是根据从多个摄像机或在整个场景中移动的单个摄像机获得的图像来构建场景的过程。然而，当前的3D视觉由于三个核心问题而受到一些问题的困扰：强调点而不是曲线和曲面；来自多个图像的信息的次优整合，以及需要更丰富的场景语义几何结构。该项目旨在通过开发处理曲线和曲面的新技术，同时集成来自多个摄像机的信息，并使用语义基元来描述场景中的对象，来纠正这三个缺点。这些发展将对现有的大量3D姿势估计和重建技术进行补充和提供宝贵的补充，特别是在人造环境等无纹理场景中。该研究的应用领域包括机器人、娱乐业、考古学、建筑学、城市建模和计量学，该项目开发了一种基于曲线曲面微分几何的端到端姿态估计和场景重建技术。首先，它开发了在姿势估计过程中使用点切线而不是仅仅使用点的必要技术，在使用已有信息减少必要对应的数量方面具有变革性。它还将确定这一领域中最小的问题。其次，这种方法将允许使用直观定义的几何方程来整合来自多个摄像头的信息，这些几何方程可以轻松扩展到三个摄像头之外，这在视觉里程计等应用中很重要。第三，开发了一种从3D曲线图进行曲面重建的技术，该技术既考虑了纹理区域中的无组织点重建，又考虑了非纹理区域中的一组新的微分光度约束。第四，它将基于图像的分组过程与同步的3D重建过程相结合，产生了一种中级表示，即一组拓扑连接的曲线碎片和表面碎片，更好地匹配了语义和功能任务的要求。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

3D vision is the process of constructing a scene from images obtained from multiple cameras or a single camera that is moving throughout the scene. However, current 3D vision suffers some problems due to three core issues: the emphasis on points rather than on curves and surfaces; sub-optimal integration of information from multiple images, and the need for richer semantic geometrical structuring of the scene. This project aims to rectify all three shortcomings by developing new technologies which work with curves and surfaces, integrate information from multiple cameras simultaneously, and use semantic primitives to describes objects in the scene. These developments will complement and present an invaluable addition to the plethora of existing techniques for 3D pose estimation and reconstruction, especially in textureless scenes such as man-made environments. Applications of this research include robotics, entertainment industry, archaeology, architecture, urban modeling, and metrology.This project develops an end-to-end technology for pose estimation and scene reconstruction using differential geometry of curves and surfaces. First, it develops the technology necessary to use point-tangents as opposed to just points in the pose estimation process, transformative in reducing the number of necessary correspondences using information that is already available. It will also identify the minimal problems in this area. Second, the approach would allow for the integration of information from many cameras using intuitively defined geometric equations that can easily extend beyond three cameras, important in applications like visual odometry. Third, it develops a technology for surface reconstruction from a 3D curve graph acting like scaffold, while also taking into account both unorganized point reconstructions in textured areas and a novel set of differential photometric constraints in textureless areas. Fourth, it integrates the image-based grouping process with a simultaneous 3D reconstruction process resulting in a mid-level representation as a topologically connected set of curve fragments and surface fragments, which better matches the requirement of semantic and functional tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.