基于多模态显性关系建模的短视频分析方法研究

随着智能移动设备的普及，每天有海量的短视频被拍摄和上传到社交媒体网站上。面对如此大规模的短视频数据，短视频分析成为了多媒体领域的一个热点研究问题。由于短视频是异构多模态信息的统一体，这些信息从不同的视角描绘着短视频的内容，因而充分且合理地利用多模态信息对短视频进行分析是十分必要的。现有短视频分析研究大多采用串联多模态特征的方法对短视频内容进行表示，并未对模态间的相关性关系进行显性的区分，使得各模态不能彼此协作增强，从而限制了对短视频内容的综合表示能力。为此，本项目拟从多模态显性关系建模的角度出发，包括一致性、互补性和协作性，对短视频分析方法展开研究。具体地，本项目将多模态显性关系建模与短视频的语义稀疏性、异步时序性以及场景结构化特性建模有机地结合在一起，拟设计基于多模态显性关系的短视频表示学习方法，来更加准确和全面地理解短视频内容，从而更好地为下游分析任务服务。

With the popularity of smart mobile devices, enormous micro-videos are shot and uploaded to social media platforms daily. Such large-scale micro video data makes micro-video analysis become a hotspot in the field of multimedia. As micro-videos are the unity of textual, visual, and acoustic modalities, which characterize the content of micro-videos from different views, thereby adequately and reasonably utilizing multi-modal information to analyze micro-videos is critical. However, most of the existing studies adopt the cascade strategy to obtain the representations of micro-videos, totally ignoring the relationships among different modalities. This makes the information of one modality cannot be enhanced by that of the other two modalities, impeding the comprehensive understanding of micro-videos. In light of this, this project intends to study the micro video analysis methods from the aspect of the multi-modal explicit relationship modeling, including the consistency, complementarity, and collaborative relationship. Specifically, by considering the characteristics of semantic sparsity, asynchronous sequence, and structured venue information, this project intends to build micro-video representation learning approaches according to three relationships. These methods can more accurately and comprehensively understand the content of the micro-video, and better serve downstream tasks.