权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

Cross-modal egocentric activity recognition and zero-shot learning

跨模式自我中心活动识别和零样本学习

基本信息

批准号：
1971464
负责人：
金额：
--
依托单位：
University of Bristol
依托单位国家：
英国
项目类别：
Studentship
财政年份：
2017
资助国家：
英国
起止时间：
2017 至无数据
项目状态：
已结题

来源：
https://gtr.ukri.org/projects?ref=studentship-1971464
关键词：
Cross modal egocentric activity recognition

项目摘要

The availability of low-cost wearable cameras has renewed the interest of first-person human activity analysis. The recognition of first-person activities has important challenges to be addressed such as, rapid changes in illuminations, significant camera motion and complex hand-object manipulations. In recent years, the advances in deep learning have influenced significantly the computer vision community, as convolutional networks gave impressive results in tasks such as, object recognition and detection, scene understanding and image segmentation. Convolutional networks have been used with success in first-person activity recognition as well. Before the emergence of deep learning the community of first-person computer vision was focused on the engineering of important egocentric features that capture properties of the first-person point of view, such as hand-object interactions and gaze. Convolutional networks allow the learning of such features automatically using big amounts of data, eliminating the need of hand-designed features. In this work, we focus on activity recognition with convolutional networks. Influenced by the recent success of multi-stream architectures, we are investigating their applicability in egocentric videos, by employing multiple modalities for training the models. An important observation that motivates us is that humans combine their senses to understand concepts of the world, such as acoustic and visual information. To this end, we propose the employment of both videos and sounds towards more accurate activity recognition. Specifically, we will investigate how shared aligned representations can be learnt using the multi-stream paradigm. Moreover, we are interested in temporal feature pooling methods to leverage information that spans over the whole video, as in many cases the whole video should be observed in order to be able to discriminate between similar actions. Our final goal is to employ these ideas in zero-shot learning. Zero-shot learning is being able to solve a task despite not having received any training examples of that task. An example is to recognize activities without having seen any video of these activities during training. This can be done by using the knowledge of trained classifiers (trained in other classes and not in the ones to be predicted by the zero-shot paradigm) and additional knowledge about the new classes.

低成本可穿戴摄像头的出现重新激发了人们对第一人称人类活动分析的兴趣。第一人称活动的识别有重要的挑战要解决，如照明的快速变化，显着的相机运动和复杂的手对象操作。近年来，深度学习的进步对计算机视觉社区产生了重大影响，因为卷积网络在对象识别和检测、场景理解和图像分割等任务中取得了令人印象深刻的结果。卷积网络也已成功用于第一人称活动识别。在深度学习出现之前，第一人称计算机视觉社区专注于重要的自我中心特征的工程设计，这些特征捕获第一人称视角的属性，例如手-物体交互和凝视。卷积网络允许使用大量数据自动学习这些特征，无需手工设计特征。在这项工作中，我们专注于卷积网络的活动识别。受多流架构最近成功的影响，我们正在研究它们在以自我为中心的视频中的适用性，通过采用多种模式来训练模型。激励我们的一个重要观察是，人类联合收割机结合他们的感官来理解世界的概念，例如听觉和视觉信息。为此，我们建议使用视频和声音来进行更准确的活动识别。具体来说，我们将研究如何共享对齐表示可以使用多流范式学习。此外，我们对时间特征池方法感兴趣，以利用跨越整个视频的信息，因为在许多情况下，应该观察整个视频，以便能够区分类似的动作。我们的最终目标是将这些想法应用于零射击学习。零触发学习是指即使没有接受过任何任务的训练示例，也能够解决该任务。一个例子是在培训期间识别活动，而无需查看这些活动的任何视频。这可以通过使用训练的分类器的知识（在其他类中训练，而不是在由零触发范例预测的类中训练）和关于新类的附加知识来完成。

项目成果

期刊论文数量（4）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

Slow-Fast Auditory Streams for Audio Recognition

DOI：
10.1109/icassp39728.2021.9413376
发表时间：
2021-03
期刊：
ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
影响因子：
0
作者：
E. Kazakos;Arsha Nagrani;Andrew Zisserman;D. Damen
通讯作者：
E. Kazakos;Arsha Nagrani;Andrew Zisserman;D. Damen

The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines