Geometric and Semantic Structures for Two- and Three-Dimensional Shape Understanding

真正的计算机视觉将提供端到端的图像分析，其中图像被分解为感兴趣的对象，这些对象被分解为部分，并且部分和对象被识别。使用图像执行集成任务（如形状生成、动画、编辑或部分匹配）需要结构感知形状处理。一个完整的形状结构包括分解成部分，理解哪些部分比其他部分更重要，并能够测量走向识别的部分的相似性。将二维或三维图像作为输入，执行精确分割以确定感兴趣的形状，提取形状结构，然后识别零件和形状的管道将代表人工视觉的基本进步。这项任务具有挑战性，因为人类的视觉感知不遵循计算规则。例如，两个形状可以都类似于第三形状，而不彼此类似。另一方面，我们对形状意义的理解为我们的几何感知增加了一个语义层面：如果有人坐在一个物体上，我们会把这个物体归类为椅子，而不管它的形状如何。任何有用的形状分析都必须明确地建模语义和几何形状之间的相互作用。该项目旨在发展形状结构的基础理论，并提供所产生的技术的鲁棒实现，同时通过对用户研究的基准测试来保持与人类语义感知的联系。Blum中轴给出了欧几里得空间中封闭区域的骨架分解。对于空间维度2和3，这些区域可以被解释为2D和3D形状，骨架模型提供了形状的低维表示。骨架，惠特尼分层集，是一个变形收缩的形状边界，捕捉完整的几何信息的边界形状。该项目将介绍中轴上的函数，其编码形状几何形状的方式允许确定形状内的部分分解和层次结构，以及部分之间的相似性，对于任何有限属的形状。在此分析的基础上，该研究将通过人类感知研究开发形状复杂性的正式措施和基准结果。最后，该项目旨在通过开发网络架构，从自然或二进制图像表示中学习形状骨架的几何形状，将新的形状结构表征与当前使用神经网络进行图像理解的方法联系起来。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

True computer vision will provide end-to-end image analysis, where images are decomposed into objects of interest, those objects are decomposed into parts, and the parts and objects are recognized. Performing integrated tasks with an image, such as shape generation, animation, editing, or partial matching, requires structure-aware shape processing. A full shape structure consists of a decomposition into parts, an understanding of which parts are more significant than others, and an ability to measure similarity of parts moving toward recognition. A pipeline that takes as input two- or three-dimensional images, performs accurate segmentation to determine shapes of interest, extracts a shape structure, then recognizes the parts and the shapes would represent a fundamental step forward in artificial vision. The task is challenging because human visual perception does not follow computational rules. For example, two shapes can both be similar to a third shape without being similar to each other. For another, our understanding of meaning of shapes adds a semantic level to our geometric perception: if someone is seated on an object, we classify that object as a chair regardless of its shape. Any useful shape analysis must explicitly model the interplay between semantics and geometric shape. This project aims to develop the foundational theory of shape structure and provide robust implementations of the resulting techniques while maintaining the connection to human semantic perception through benchmarking to user studies.The Blum medial axis gives a skeletal decomposition of a closed region in Euclidean space. For spatial dimensions 2 and 3, these regions can be interpreted as 2D and 3D shapes, with the skeletal model providing a lower-dimensional representation of the shape. The skeleton, a Whitney stratified set, is a deformation retract of the shape boundary that captures complete geometric information about the boundary of the shape. This project will introduce functions on the medial axis that encode shape geometry in a way that allows for the determination of a parts decomposition and hierarchy within a shape, as well as similarity between parts, for shapes of any finite genus. Based on that analysis, the research will develop formal measures of shape complexity and benchmark results through human perception studies. Finally, the project aims to connect the new shape structure characterization to current approaches using neural networks for image understanding by developing network architectures that learn the geometry of a shape skeleton from its natural or binary image representation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.