随着互联网中带有图片的多模态文档数据（如新闻、博客）的爆炸式增长，包含文本和图片的图文摘要正成为人们更好更快获取信息的迫切需要。针对当前文本摘要方法中图片信息缺失的现状，利用递归注意力神经网络有效解决序列生成和分类问题的优点，本项目建立新的图文摘要方法，研究结合深度学习的图文摘要关键技术，主要有三个创新点：提出基于双层注意力模型的图文对齐模型，将句子与图片对齐，词与像素对齐，实现图文对齐关系发掘，用于图文摘要产生和排版；提出基于双层注意力编码-解码模型的生成式图文摘要方法，同时考虑对图片的注意力和对文本的注意力，先解码产生句表示，再解码产生词表示，选择图片并排版，实现图文摘要生成；提出基于递归神经网络的抽句式图文摘要方法，以对图片信息的覆盖度和对文本信息的覆盖度衡量句子重要度，选取重要句子和图片并排版，实现图文摘要抽取。本项目提供新的图文摘要范式和图文对齐方法，具有重要的科研价值和应用价值

With rapid growth of multi-modal document data with pictures such as news and blogs, text-picture summaries containing text and pictures have now become an emergency need for people better and faster getting information. To deal with the lack of picture information in traditional text summarization methods, this project builds a new text-picture summarization method based on attentional recurrent neural network, and studies the key image-text summarization techniques by combining deep learning and picture information. The main innovation points are as follows. 1) We propose text-picture alignment methods based on the hierarchical attentional model, which aligning sentences with pictures and aligning words with pixels. The model mines the hidden text-picture aligning relationships which are used for summary creation and summary alignment. 2) We propose generative the text-picture summarization method based on the hierarchical attentional Encoder-Decoder model by combining both the attention to pictures and the attention to text. The model generates text-picture summaries by firstly generating the sentence representations, secondly generating word representations, thirdly selecting pictures, and finally rearranging the generated texts and pictures. 3) We propose the extractive text-picture summarization method based on the recurrent neural network. The model computes the importance of the sentence using its coverage of the picture information and its coverage of text information, and selects and rearranges important sentences and pictures to form extractive summaries. This project provides a new paradigm of multi-modal summarization and a text-picture alignment method, and thus has significant scientific values and practical values.