基于场景文字的图像描述生成关键技术研究

图像描述生成任务旨在生成与图像内容相符且逻辑合理的文本描述。其横跨视觉和语言两大模态，可应用于机器人聊天、视障人群辅助等场景，具有重要的研究价值和广阔的应用前景。然而，现有算法大多不具备读取图像中场景文字的能力，导致生成的描述文本容易遗漏图像中的重要信息。为此，本项目拟以含有场景文字的自然场景图像为研究对象，融合计算机视觉、自然语言处理等领域知识，借助预训练大模型、自监督学习、注意力机制等最新技术手段，系统地开展基于场景文字的图像描述生成研究。具体地，拟提出基于视觉关系和语义关系的多模态表征学习方法、文本描述中场景文字的定位与表征选取方法、基于对比学习的场景文字表征优化方法和自回归与非自回归融合的语言模型等，以提高多模态表示的表征能力及描述生成模型的鲁棒性。本项目有望突破现有图像描述生成技术的瓶颈，为多模态内容的知识分析、智能生成等研究和应用提供理论支持和技术支撑。

Image captioning aims to generate textual descriptions that accurately reflect the content of an image in a logical and coherent manner. It bridges the two modalities of vision and language and has a wide range of applications, such as in robot chatting and providing visual aids for people with visual impairments. Therefore, it has important research value and promising application prospects. However, most existing approaches do not have the ability to read scene text in images, resulting in the lack of important information from the image in the description. To address this, this project aims to integrate knowledge from fields such as computer vision and natural language processing, and systematically conduct research on text-based image captioning using techniques such as pretrained large models, self-supervised learning, and attention mechanisms. Specifically, we plan to propose a multi-modal representation learning method based on visual and semantic relationships, a method for locating and selecting representations of scene text in text description, a method for optimizing scene text representation based on contrastive learning, and a language model that combines autoregressive and non-autoregressive methods to improve the representation ability of multi-modal inputs and the robustness of captioning models. This project is expected to break through the bottleneck of current image captioning technology and provide both theoretical and technical support for research and applications in knowledge analysis and artificial intelligent generation of multimodal content.