知识驱动的视频文本描述生成

随着网络视频应用的不断普及和视频数量的爆发式增长，网络用户能便利地浏览海量的视频。但视频数据本身所具有的复杂特性也给自动分析和理解这些网络视频带来了巨大的挑战。为了探索更智能的视频内容分析方法，本项目重点研究知识驱动的视频文本描述生成方法。主要研究工作将从如下三个方面展开：（1）在知识学习方面，我们研究基于实体抽取、属性抽取和关系抽取等技术从第三方非结构化（或半结构化）数据集中挖掘属性知识和关系知识。拟采用深度表示学习技术把视觉属性知识以及相对关系知识嵌入到语义空间得到知识的实数化向量表示。（2）在视频内容理解方面，我们研究基于深度学习技术并结合知识表示提取视频中的重要片段。拟采用深度排序神经网络学习视频片段的高层语义特征表示。（3）在视频文本描述生成方面，我们研究基于属性学习网络和知识的关注度模型的文本描述生成方法。拟采用结构化学习方法建模视频中的物体、行为和场景等视觉对象之间的联系。

With the rapid development of the video application, huge number of the videos have been uploaded on the Internet. Web users can conveniently access and browse these videos. However, it is also a very challenging task to automatically analyze and understand the semantic content contained in the complex video data. To explore more effective method for video content analysis, we will focus on the knowledge-driven video captioning in this project. The research plan has three important steps: (1) In knowledge representation, we mainly study mining attribute and relation knowledge from third-part un-structured/semi-structured data based on entity extraction, attribute extraction and relation extraction methods. We will adopt the deep representation learning algorithm to embed the visual attribute knowledge and the relative relation knowledge into a common semantic space. (2) In the video content understanding, we mainly study video highlight extraction based on deep learning and knowledge representation. We will adopt deep ranking model to learn the high-level semantic features of the video highlight part. (3) In the video description generation, we mainly study sentence generation method based on attribute networks and knowledge sensitive attention model. We will adopt structured prediction algorithm to model the visual relations among objects, actions and scenes in the video.