Cross-modal egocentric activity recognition and zero-shot learning

低成本可穿戴相机的出现重新燃起了人们对第一人称人类活动分析的兴趣。第一人称活动的识别有许多重要的挑战需要解决，如光照的快速变化、显著的相机运动和复杂的手-物体操作。近年来，深度学习的进步对计算机视觉社区产生了重大影响，卷积网络在物体识别和检测、场景理解和图像分割等任务中取得了令人印象深刻的成果。卷积网络在第一人称活动识别方面也取得了成功。在深度学习出现之前，第一人称计算机视觉社区专注于捕获第一人称视角属性的重要自我中心特征的工程，例如手-对象交互和凝视。卷积网络允许使用大量数据自动学习这些特征，从而消除了手工设计特征的需要。在这项工作中，我们专注于卷积网络的活动识别。受最近多流架构成功的影响，我们正在研究它们在以自我为中心的视频中的适用性，通过采用多种模式来训练模型。一个激励我们的重要观察是，人类结合他们的感官来理解世界的概念，比如声音和视觉信息。为此，我们建议使用视频和声音来实现更准确的活动识别。具体来说，我们将研究如何使用多流范式学习共享对齐表示。此外，我们对时间特征池化方法感兴趣，以利用跨越整个视频的信息，因为在许多情况下，为了能够区分相似的动作，应该观察整个视频。我们的最终目标是将这些理念运用到零射击学习中。零射击学习是指在没有接受任何训练的情况下能够解决任务。一个例子是在训练过程中，在没有看到任何视频的情况下识别这些活动。这可以通过使用训练过的分类器的知识（在其他类中训练，而不是在由零射击范式预测的分类器中训练）和关于新类的额外知识来完成。

The availability of low-cost wearable cameras has renewed the interest of first-person human activity analysis. The recognition of first-person activities has important challenges to be addressed such as, rapid changes in illuminations, significant camera motion and complex hand-object manipulations. In recent years, the advances in deep learning have influenced significantly the computer vision community, as convolutional networks gave impressive results in tasks such as, object recognition and detection, scene understanding and image segmentation. Convolutional networks have been used with success in first-person activity recognition as well. Before the emergence of deep learning the community of first-person computer vision was focused on the engineering of important egocentric features that capture properties of the first-person point of view, such as hand-object interactions and gaze. Convolutional networks allow the learning of such features automatically using big amounts of data, eliminating the need of hand-designed features. In this work, we focus on activity recognition with convolutional networks. Influenced by the recent success of multi-stream architectures, we are investigating their applicability in egocentric videos, by employing multiple modalities for training the models. An important observation that motivates us is that humans combine their senses to understand concepts of the world, such as acoustic and visual information. To this end, we propose the employment of both videos and sounds towards more accurate activity recognition. Specifically, we will investigate how shared aligned representations can be learnt using the multi-stream paradigm. Moreover, we are interested in temporal feature pooling methods to leverage information that spans over the whole video, as in many cases the whole video should be observed in order to be able to discriminate between similar actions. Our final goal is to employ these ideas in zero-shot learning. Zero-shot learning is being able to solve a task despite not having received any training examples of that task. An example is to recognize activities without having seen any video of these activities during training. This can be done by using the knowledge of trained classifiers (trained in other classes and not in the ones to be predicted by the zero-shot paradigm) and additional knowledge about the new classes.