Connecting Vision and Language in Instructional Videos through Human Gaze

点击翻译按钮获取中文摘要

The project unfortunately only last for half a year, however fruitful outcomes have still been made. In the research plan, we originally proposed to collect the world’s first instructional video dataset with synchronized language and gaze. Since the purchase of wearable equipment has not been ready due to the low productivity casued by Covid, in this fiscal year we collected a preliminary dataset using a similar setting, i.e., we recorded user's eye-tracking data on a fixed screen, containing a person’s gaze when he is watching and describing the video. In this way we could still analyze the dense connection between video, gaze and language. In total we have collected 2.5 hours of multi-modal data on both the video datasets. The audio data were collected in 15 different languages. We are still working on the publication of this dataset. While the total amount of collected data is limited, it is a very good start.Another part of research plan is to develop vision language algorithm aiming at publishing top-tier academic papers. Addressing the problem that we do not have enough data for model training at the moment, I developed few-shot based action recognition algorithm, which can significantly improve the model’s ability when given only less than 5 annotated samples. The paper has been accepted by top tier academic conference ECCV 2022.