DEFORM: Large Scale Shape Analysis of Deformable Models of Humans

最近，计算机视觉正在见证一种范式的转变。标准的鲁棒特征，如尺度不变特征变换（SIFT）、定向梯度直方图（hog）等，通过应用深度卷积神经网络（DCNNs）被可学习滤波器取代。此外，对于涉及可变形对象（如人体/面部/手等）的应用（例如，检测、跟踪、识别等），传统的基于统计或物理的可变形模型与DCNNs相结合，效果非常好。目前的进展是由于大数据时代大量复杂的视觉数据，这些数据主要通过互联网通过Youtube、Flickr和谷歌Images等网络服务传播。后者导致了大型数据库（如ImageNet、Microsoft COCO和300W等）的发展，这些数据库由“在野外”捕获的视觉数据组成。此外，科学界和工业界已经承担了大规模的标注任务。例如，我和我的团队付出了巨大的努力，针对大量的面部地标标注了超过30K的面部图像和500K的视频帧。COCO团队已经注释了数千张身体图像，包括身体关节等。以上所有注释一般都是指对象和/或其片段的一组稀疏部分，这些部分可以由人类进行注释（例如，通过众包）。为了使下一步自动理解场景，特别是人类和他们的行为，社区需要获取3D密集信息。尽管收集2D强度图像现在是一个相对容易和廉价的过程，但收集可变形物体（如人类及其（身体）部位）的高分辨率3D扫描仍然是一个昂贵和费力的过程。这就是为什么在收集3D人脸，头部，手，身体等的大规模数据库方面所做的努力非常有限的主要原因。在DEFORM中，我建议进行大规模的高分辨率人体4D序列收集。此外，我提出了新的研究方向，以便提供关于二维强度“野外”图像与可变形物体形状（特别是人类及其部位）的密集3D结构之间的对应关系的高质量注释。建立密集的2d到3d对应关系可以毫不费力地解决许多图像级任务，如地标（部分）定位，密集语义部分分割，变形估计（即行为）等。

Recently, computer vision is witnessing a paradigm shift. Standard robust features, such as Scale Invariant Feature Transform (SIFT), Histogram of Oriented Gradienst (HoGs), etc., are replaced by learnable filters via the application of Deep Convolutional Neural Networks (DCNNs). Furthermore, for applications (e.g., detection, tracking, recognition, etc.) that involve deformable objects, such as human bodies/faces/hands etc., traditional statistical or physics-based deformable models are combined with DCNNs with very good results. The current progress is made due to the abundance of complex visual data in the Big Data era, spread mostly through the Internet via web services such as Youtube, Flickr, and Google Images. The latter has led to the development of huge databases (such as ImageNet, Microsoft COCO, and 300W, etc.) consisting of visual data captured "in-the wild". Furthermore, the scientific and industrial community has undertaken large-scale annotation tasks. For example, me and my group have made huge efforts to annotate over 30K facial images and 500K video frames with regards to a large number of facial landmarks. The COCO team has annotated thousands of body images with regards to body joints, etc. All the above annotations generally refer to a set of sparse parts of objects and/or their segments, which can be annotated by humans (e.g., through crowd sourcing). In order to make the next step in automatic understanding of a scene in general, and humans and their actions, in particular, the community needs to acquire 3D dense information. Even though the collection of 2D intensity images is now a relatively easy and inexpensive process, the collection of high-resolution 3D scans of deformable objects, such as humans and their (body) parts, still remains an expensive and laborious process. This is the principal reason why very limited efforts have been made in collecting large-scale databases of 3D faces, heads, hands, bodies, etc.In DEFORM, I propose to perform large-scale collection of high-resolution 4D sequences of humans. Furthermore, I propose new lines of research in order to provide high quality annotations regarding the correspondences between the 2D intensity "in-the-wild" images and the dense 3D structure of deformable objects' shapes and in particular of humans and their parts. Establishing dense 2D-to-3D correspondences can effortlessly solve many image-level tasks such as landmark (part) localisation, dense semantic part segmentation, estimation of deformations (i.e., behaviour), etc.