RI:Small: Learning shape features with deep neural networks

本项目研究如何利用深度神经网络从图像中有效地学习形状特征。人们普遍认为，深度神经网络从图像中学习的特征包括物体的纹理、颜色和形状。虽然学习特征的可视化显示在深度学习的过程中提取了对象的轮廓，但我们的初步结果提供了明确的论点，即当前的深度神经网络不能很好地捕获2D形状特征。这个项目开发了一个用深度神经网络有效学习形状特征的框架。这项研究为计算机视觉中的一个核心问题带来了新的见解：形状理解，它涉及计算机视觉中的许多子领域，从低层任务(如分割和图像统计)到高级任务(如图像中的视觉检索和目标检测)。该项目包括计划将研究成果直接应用于生物多样性研究(物种识别)等应用。该项目还包括高中生和本科生参与研究。该项目进行了理论和实验研究，以更好地理解为什么目前的深度神经网络不能很好地捕获形状特征。然后，针对形状特征，提出了两种新的学习策略：(1)限制卷积神经网络的滤波学习，使其更聚焦于轮廓；(2)设计特殊的深度神经网络结构来学习形状表示。该项目设计了用于基于轮廓的形状分类的圆形顺序网络，该网络自然地编码轮廓上下文信息，同时隐式地执行轮廓匹配。它还将这些网络扩展到草图，草图由闭合和开放的轮廓组成。注意力模型研究形状，以分析部件在形状表示中的角色，从而进一步改进形状匹配和识别算法。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project investigates how to effectively learn shape features with deep neural networks from images. It has been commonly believed that features learned by deep neural networks from images include texture, color, and shape of objects. Although visualizations of learned features demonstrate that contours of objects are extracted in the process of deep learning, our preliminary results provide clear arguments that 2D shape features are not well captured by current deep neural networks. This project develops a framework for effective learning of shape features with deep neural networks. The research brings new insights to a core problem in computer vision: shape understanding, which relates to many subfields in computer vision ranging from low-level tasks, such as segmentation and image statistics, to high-level ones, such as visual retrieval and object detection in images. The project includes plan to deploy the research results directly to applications such as biodiversity study (species recognition). The project also involves high school students and undergraduates in research.This project conducts both theoretical and experimental research to gain better understanding why shape features are not well captured by current deep neural networks. Then it develops new learning strategies specifically targeted for shape features by following two main alternatives: (1) constraining the filter learning for Convolutional Neural Networks so that they are more contour focused, and (2) designing special structures of Deep Neural Networks for learning shape representation. The project designs circular sequential networks for silhouette-based shape classification, which encode naturally contour context information while implicitly performing contour matching. It also extends these networks to sketches, which are composed of both closed and open contours. Attention models are investigated on shapes to analyze roles of parts in shape representations so as to improve further shape matching and recognition algorithms.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.