CAREER: Toward Spatial-Temporal Architectures with Deformable and Interpretable Convolutions

人工神经网络已成功地应用于视觉图像的分析。该项目的目标是建立一个能够自动缩放和变形的卷积神经网络(CNN)，以便能够对对象的大小和位置保持不变。目前，如果不在重新缩放的图像上进行训练，CNN甚至不能在重新缩放的图像上表现良好。这会导致模型中的大量冗余和架构不必要的过度复杂。这个项目探索了从图像和视频中的可视对象自动计算出正确的缩放比例以及其他转换的方法。所提出的方法还将使卷积神经网络更容易解释，并减少训练网络所需的数据量。除了正常的计算机视觉基准之外，研究团队还与合作伙伴评估了该方法，以将技术应用于不同的应用，如林业和肿瘤细胞形态学，该项目的教育目标包括开发一种新的？你看到的是什么？你得到了什么？(所见即所得)深度学习工具箱，使没有太多编程和数学技能的人能够利用深度学习进行数据分析。研究小组还计划接触高中和社区大学，向100多名学生介绍深度学习和视觉对象识别。该研究开发了自动缩放和变形的时空CNN，从而能够简明地表示对象识别模型，这些模型在训练集中看不到的不变和等变变换下具有更好的泛化能力。该项目探索了新的自动缩放和多变形卷积网络结构，该结构利用参数运动场来自动定位每个卷积过滤器的视觉对象的正确变形。为了从视频中学习运动场，研究团队使用暹罗卷积-反卷积网络在两个连续的帧中预测边界，并利用输出到输出的反馈环路来推断边界运动。研究小组将这种方法应用于视频分割，并使用它为运动场的弱监督学习生成注释。该方法在几个注释有限的任务上进行了评估，如视频分割、多目标跟踪以及在不可见变形下的视频中的对象分类和检测。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Artificial neural networks have successfully been applied to analyzing visual imagery. The goal of this project is to build a convolutional neural network (CNN) that can scale and deform automatically in order to be able to be invariant to object size and pose.Currently, CNNs cannot even perform well on an image rescaled twice or half as large, if not trained on the re-scaled image. This leads to a lot of redundancies in the model and unnecessary over-complication of the architecture. This project explores approaches to automatically figure out the correct scaling, as well as other transformations, from visual objects in images and videos. The proposed methods will also make convolutional neural networks easier to interpret, and to reduce the amount of data needed to train a network.Besides normal computer vision benchmarks, the research team evaluates the approach with collaborations to apply the technologies to different applications, such as forestry and tumor-cell morphology, The educational goal of this project involves developing a new ?what-you-see-is-what-you-get? (WYSIWYG) deep learning toolbox that enables people without much programming and mathematical skills to utilize deep learning for data analysis. The research team also plans to outreach to high schools and community colleges to introduce more than 100 students to deep learning and visual object recognition. This research develops spatial-temporal CNNs that scale and deform automatically, hence able to concisely represent object recognition models that generalize better under invariant and equivariant transformations unseen in the training set. The project explores novel auto-scaling and multi-deformable convolutional network architectures that utilize parametric motion fields to automatically locate the correct deformations of a visual object for each convolutional filter. In order to learn the motion fields from video, the research team uses a Siamese convolutional-deconvolutional network predicting boundaries in two consecutive frames, and utilizes an output-to-output feedback loop to deduce boundary motion. The research team applies this approach to video segmentation and uses it to generate annotations for a weakly supervised learning of the motion fields. The approach is evaluated on several tasks with limited annotations, such as video segmentation, multi-target tracking and object classification and detection in videos under unseen deformations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.