主流对话系统利用的数据资源和事实依据以自然语言单一模态为主，无法满足现实多模态应用需求。视觉聊天机器人是图像与自然语言理解交叉的热门方向，但目前研究类型有限，局限于非知识性问答和“猜猜看”游戏。功能性和类人性是机器人的重要技术发展趋势。为了实现更为知性的对话系统，本项目提出围绕绘本教学过程，在限定领域内研究基于视觉-语义多模态知识图谱的情感嵌入式对话生成算法。首先，项目收集该任务的视觉对话数据集，不失一般性，研究视觉-语义融合的多模态知识图谱构建方法。其次，为了准确理解用户输入并生成期望回复，项目研究对话意图图谱和情感自动分析与嵌入机制。最后，基于前述两方面工作，项目研究主题共享、多形式对话穿插的语言生成机制，以实现更自然的人机交互。本项目在算法上实现知识图谱、视觉计算、自然语言对话生成等方向的交叉融合，对于发展多模态应用场景下新型对话系统、实现更高智能的聊天机器人具有重要的科学意义。

The data resources and facts used by mainstream dialogue systems are mainly based on natural language documents, which cannot meet the requirements of realistic multi-modal applications. Visual chatbot is a popular interdisciplinary research direction of computer vision and natural language understanding. However, at present, its research types are limited to non-knowledge question-answer and "guess what" game. Functional and humanoid robot is an important technology development trend. In order to study more intellectual dialogue system, this project proposes to study emotional embedded dialogue generation algorithm based on the visual-semantic multi-modal knowledge graph from three aspects within the limited field of picture books teaching. First, the project collects visual dialogue data sets of featured tasks and devotes itself to the research and construction of a visual-semantic fusion multimodal knowledge graph. Secondly, in order to accurately understand user's input and generate expected responses, the project proposes to study automatic analysis and embedding mechanism of dialogue intention graph and emotional information. Finally, on the basis of the first two work, this project studies the language generation mechanism for multiform interactions like chat and QA with shared conversation topics, so as to achieve more natural human-computer interaction. The project organically combines knowledge graph computing, visual computing, natural language generation in algorithm aspect, which is of great scientific significance for the development of a new type of dialogue system under multi-modal application scenarios and the realization of a more intelligent chatbot.