DEFORM: Large Scale Shape Analysis of Deformable Models of Humans

最近，计算机视觉正在见证一种范式的转变。通过应用深度卷积神经网络(DCNN)将尺度不变特征变换(SIFT)、定向梯度直方图(HOGS)等标准稳健特征替换为可学习的滤波器。此外，对于应用(例如，检测、跟踪、识别等)在涉及人体/人脸/手等可变形物体的情况下，将传统的基于统计或物理的可变形模型与DCNN相结合，取得了很好的效果。目前的进展归功于大数据时代丰富的复杂视觉数据，这些数据主要通过YouTube、Flickr和Google Images等网络服务通过互联网传播。后者导致了大型数据库的发展(如ImageNet、Microsoft Coco和300W等)。由在野外捕获的可视数据组成。此外，科学界和工业界还承担了大规模的注释任务。例如，我和我的团队付出了巨大的努力，针对大量的面部地标标注了超过30K的面部图像和500K的视频帧。Coco团队已经注释了数千个关于身体关节等的身体图像。所有上述注释通常指的是对象和/或其片段的一组稀疏部分，这些部分可以由人类注释(例如，通过众包)。为了进行下一步对场景的自动理解，特别是对人类及其行为的理解，社区需要获取3D密集信息。尽管2D强度图像的收集现在是一个相对容易和便宜的过程，但收集可变形物体的高分辨率3D扫描，如人体及其(身体)部分，仍然是一个昂贵和费力的过程。这就是为什么在收集大规模的3D人脸、头部、手、身体等数据库方面所做的努力非常有限的主要原因。在Deform中，我建议大规模收集高分辨率的4D人体序列。此外，我还提出了新的研究路线，以提供高质量的注释，说明2D强度“野外”图像与可变形对象形状的密集3D结构之间的对应关系，特别是人类及其部分的形状。建立密集的2D到3D对应可以毫不费力地解决许多图像级任务，例如地标(部分)定位、密集语义部分分割、变形(即行为)估计等。

Recently, computer vision is witnessing a paradigm shift. Standard robust features, such as Scale Invariant Feature Transform (SIFT), Histogram of Oriented Gradienst (HoGs), etc., are replaced by learnable filters via the application of Deep Convolutional Neural Networks (DCNNs). Furthermore, for applications (e.g., detection, tracking, recognition, etc.) that involve deformable objects, such as human bodies/faces/hands etc., traditional statistical or physics-based deformable models are combined with DCNNs with very good results. The current progress is made due to the abundance of complex visual data in the Big Data era, spread mostly through the Internet via web services such as Youtube, Flickr, and Google Images. The latter has led to the development of huge databases (such as ImageNet, Microsoft COCO, and 300W, etc.) consisting of visual data captured "in-the wild". Furthermore, the scientific and industrial community has undertaken large-scale annotation tasks. For example, me and my group have made huge efforts to annotate over 30K facial images and 500K video frames with regards to a large number of facial landmarks. The COCO team has annotated thousands of body images with regards to body joints, etc. All the above annotations generally refer to a set of sparse parts of objects and/or their segments, which can be annotated by humans (e.g., through crowd sourcing). In order to make the next step in automatic understanding of a scene in general, and humans and their actions, in particular, the community needs to acquire 3D dense information. Even though the collection of 2D intensity images is now a relatively easy and inexpensive process, the collection of high-resolution 3D scans of deformable objects, such as humans and their (body) parts, still remains an expensive and laborious process. This is the principal reason why very limited efforts have been made in collecting large-scale databases of 3D faces, heads, hands, bodies, etc.In DEFORM, I propose to perform large-scale collection of high-resolution 4D sequences of humans. Furthermore, I propose new lines of research in order to provide high quality annotations regarding the correspondences between the 2D intensity "in-the-wild" images and the dense 3D structure of deformable objects' shapes and in particular of humans and their parts. Establishing dense 2D-to-3D correspondences can effortlessly solve many image-level tasks such as landmark (part) localisation, dense semantic part segmentation, estimation of deformations (i.e., behaviour), etc.