Geometric and Semantic Structures for Two- and Three-Dimensional Shape Understanding

真正的计算机视觉将提供端到端的图像分析，其中图像被分解为感兴趣的对象，这些对象被分解为部件，然后识别部件和对象。对图像执行集成任务，如形状生成、动画、编辑或部分匹配，需要结构感知形状处理。一个完整的形状结构包括分解成各个部分，理解哪些部分比其他部分更重要，以及测量走向识别的部分的相似性的能力。将二维或三维图像作为输入，进行精确分割以确定感兴趣的形状，提取形状结构，然后识别零件和形状的管道将代表人工视觉的基本进步。这项任务具有挑战性，因为人类的视觉感知并不遵循计算规则。例如，两个形状都可以与第三个形状相似，但彼此不相似。另一方面，我们对形状意义的理解为我们的几何感知增加了一个语义层面：如果有人坐在一个物体上，我们就把这个物体归类为椅子，而不管它的形状如何。任何有用的形状分析都必须明确地对语义和几何形状之间的相互作用进行建模。该项目旨在发展形状结构的基础理论，并提供所得到的技术的健壮实现，同时通过对用户研究的基准测试保持与人类语义感知的联系。布卢姆中轴给出了欧几里得空间中封闭区域的骨架分解。对于空间维度2和3，这些区域可以被解释为2D和3D形状，骨骼模型提供了形状的低维表示。骨架是一个惠特尼分层集，是形状边界的变形缩回，它捕获了关于形状边界的完整几何信息。该项目将引入中间轴上的函数，这些函数以一种允许确定形状内的部件分解和层次结构的方式编码形状几何，以及部件之间的相似性，适用于任何有限属的形状。基于这一分析，该研究将通过人类感知研究开发形状复杂性的正式测量方法和基准结果。最后，该项目旨在通过开发从自然或二值图像表示中学习形状骨架几何的网络架构，将新的形状结构表征与使用神经网络进行图像理解的当前方法联系起来。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

True computer vision will provide end-to-end image analysis, where images are decomposed into objects of interest, those objects are decomposed into parts, and the parts and objects are recognized. Performing integrated tasks with an image, such as shape generation, animation, editing, or partial matching, requires structure-aware shape processing. A full shape structure consists of a decomposition into parts, an understanding of which parts are more significant than others, and an ability to measure similarity of parts moving toward recognition. A pipeline that takes as input two- or three-dimensional images, performs accurate segmentation to determine shapes of interest, extracts a shape structure, then recognizes the parts and the shapes would represent a fundamental step forward in artificial vision. The task is challenging because human visual perception does not follow computational rules. For example, two shapes can both be similar to a third shape without being similar to each other. For another, our understanding of meaning of shapes adds a semantic level to our geometric perception: if someone is seated on an object, we classify that object as a chair regardless of its shape. Any useful shape analysis must explicitly model the interplay between semantics and geometric shape. This project aims to develop the foundational theory of shape structure and provide robust implementations of the resulting techniques while maintaining the connection to human semantic perception through benchmarking to user studies.The Blum medial axis gives a skeletal decomposition of a closed region in Euclidean space. For spatial dimensions 2 and 3, these regions can be interpreted as 2D and 3D shapes, with the skeletal model providing a lower-dimensional representation of the shape. The skeleton, a Whitney stratified set, is a deformation retract of the shape boundary that captures complete geometric information about the boundary of the shape. This project will introduce functions on the medial axis that encode shape geometry in a way that allows for the determination of a parts decomposition and hierarchy within a shape, as well as similarity between parts, for shapes of any finite genus. Based on that analysis, the research will develop formal measures of shape complexity and benchmark results through human perception studies. Finally, the project aims to connect the new shape structure characterization to current approaches using neural networks for image understanding by developing network architectures that learn the geometry of a shape skeleton from its natural or binary image representation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.