Connecting Vision and Language in Instructional Videos through Human Gaze

该项目只持续了半年，但仍然取得了丰硕的成果。在研究计划中，我们最初提出收集世界上第一个具有同步语言和凝视的教学视频数据集。由于新型冠状病毒疫情导致生产力下降，导致可穿戴设备的采购尚未准备就绪，因此我们于本财政年度使用类似的设置收集了初步数据集，即，我们在一个固定的屏幕上记录了用户的眼球跟踪数据，包括一个人在观看和描述视频时的目光。通过这种方式，我们仍然可以分析视频，凝视和语言之间的紧密联系。我们总共在两个视频数据集上收集了2.5小时的多模态数据。音频数据以15种不同的语言收集。我们仍在努力发布这个数据集。虽然收集到的数据总量有限，但这是一个很好的开始，研究计划的另一部分是开发视觉语言算法，目标是发表顶级学术论文。针对目前模型训练数据不足的问题，本文提出了基于少镜头的动作识别算法，该算法在给定少于5个标注样本的情况下，能够显著提高模型的识别能力。该论文已被顶级学术会议ECCV 2022接受。

The project unfortunately only last for half a year, however fruitful outcomes have still been made. In the research plan, we originally proposed to collect the world’s first instructional video dataset with synchronized language and gaze. Since the purchase of wearable equipment has not been ready due to the low productivity casued by Covid, in this fiscal year we collected a preliminary dataset using a similar setting, i.e., we recorded user's eye-tracking data on a fixed screen, containing a person’s gaze when he is watching and describing the video. In this way we could still analyze the dense connection between video, gaze and language. In total we have collected 2.5 hours of multi-modal data on both the video datasets. The audio data were collected in 15 different languages. We are still working on the publication of this dataset. While the total amount of collected data is limited, it is a very good start.Another part of research plan is to develop vision language algorithm aiming at publishing top-tier academic papers. Addressing the problem that we do not have enough data for model training at the moment, I developed few-shot based action recognition algorithm, which can significantly improve the model’s ability when given only less than 5 annotated samples. The paper has been accepted by top tier academic conference ECCV 2022.