Geometric and Semantic Structures for Two- and Three-Dimensional Shape Understanding

真正的计算机视觉将提供端到端的图像分析，其中图像被分解成感兴趣的对象，这些对象被分解成部分，并且部分和对象被识别。对图像执行集成任务，如形状生成、动画、编辑或部分匹配，需要结构感知的形状处理。完整的形状结构包括分解成多个部分，了解哪些部分比其他部分更重要，以及衡量各部分的相似性以便于识别的能力。一条管道将二维或三维图像作为输入，执行准确的分割以确定感兴趣的形状，提取形状结构，然后识别部件和形状，这将代表着人工视觉向前迈出的基本一步。这项任务具有挑战性，因为人类的视觉感知不遵循计算规则。例如，两个形状可以都类似于第三个形状，而不是彼此相似。另一方面，我们对形状意义的理解为我们的几何知觉增加了一个语义层面：如果有人坐在一个物体上，我们会将该物体归类为椅子，而不考虑它的形状。任何有用的形状分析都必须明确地模拟语义和几何形状之间的相互作用。该项目旨在发展形状结构的基础理论，并提供所产生的技术的健壮实现，同时通过对用户研究的基准测试保持与人类语义感知的联系。Blum中轴给出了欧几里德空间中闭合区域的骨架分解。对于空间维度2和3，这些区域可以被解释为2D和3D形状，骨骼模型提供形状的低维表示。骨架是惠特尼分层集，是形状边界的变形缩回，它捕获了有关形状边界的完整几何信息。这个项目将在中轴上引入编码形状几何的函数，允许确定形状内的部件分解和层次，以及部件之间的相似性，对于任何有限亏格的形状。基于这一分析，这项研究将开发形状复杂性的正式测量方法，并通过人类感知研究对结果进行基准测试。最后，该项目旨在将新的形状结构表征与当前使用神经网络进行图像理解的方法联系起来，通过开发从自然或二进制图像表示中学习形状骨架的几何图形的网络结构。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

True computer vision will provide end-to-end image analysis, where images are decomposed into objects of interest, those objects are decomposed into parts, and the parts and objects are recognized. Performing integrated tasks with an image, such as shape generation, animation, editing, or partial matching, requires structure-aware shape processing. A full shape structure consists of a decomposition into parts, an understanding of which parts are more significant than others, and an ability to measure similarity of parts moving toward recognition. A pipeline that takes as input two- or three-dimensional images, performs accurate segmentation to determine shapes of interest, extracts a shape structure, then recognizes the parts and the shapes would represent a fundamental step forward in artificial vision. The task is challenging because human visual perception does not follow computational rules. For example, two shapes can both be similar to a third shape without being similar to each other. For another, our understanding of meaning of shapes adds a semantic level to our geometric perception: if someone is seated on an object, we classify that object as a chair regardless of its shape. Any useful shape analysis must explicitly model the interplay between semantics and geometric shape. This project aims to develop the foundational theory of shape structure and provide robust implementations of the resulting techniques while maintaining the connection to human semantic perception through benchmarking to user studies.The Blum medial axis gives a skeletal decomposition of a closed region in Euclidean space. For spatial dimensions 2 and 3, these regions can be interpreted as 2D and 3D shapes, with the skeletal model providing a lower-dimensional representation of the shape. The skeleton, a Whitney stratified set, is a deformation retract of the shape boundary that captures complete geometric information about the boundary of the shape. This project will introduce functions on the medial axis that encode shape geometry in a way that allows for the determination of a parts decomposition and hierarchy within a shape, as well as similarity between parts, for shapes of any finite genus. Based on that analysis, the research will develop formal measures of shape complexity and benchmark results through human perception studies. Finally, the project aims to connect the new shape structure characterization to current approaches using neural networks for image understanding by developing network architectures that learn the geometry of a shape skeleton from its natural or binary image representation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.