Recognition and Description of Diagrammatic Content in Images Using Neural Networks

这个博士项目的目标是所有类型图像中的图解内容，这是一个图像分析和图像描述领域，迄今为止还没有看到与描述现实世界内容（如人、动物、无生命物体）和文本内容相同程度的关注和成功。图解图像内容被近似地定义为由几何形状组成的线条图。该项目将着眼于几个日益复杂的子任务：(1)定位图像中的图解内容；(2)将图解内容分解为组成部分；(3)向抽象图表示的映射；(4)生成图的自然语言描述。除了单独查看每个任务之外，我们还将查看两个或多个任务组合的集成解决方案。神经网络方法和混合神经网络方法将是所有解决方案的核心。确定什么样的描述适合什么样的上下文（例如用户类型、期望的抽象级别或图表的大小）将是研究的重要组成部分。总之，博士学位的主要目标是：1。开发基于神经网络的方法，生成图结构的抽象表示和图像中图内容的自然语言描述；2 .创建包含图表、抽象表示和描述的成对图像数据集，用于开发、训练和评估方法，并作为其他研究人员的重要数据资源公开发布；制定全面的评估策略，以促进对已开发的方法进行全面的评估，包括通过自动手段和涉及BPS参与者。

This PhD project is aimed at diagrammatic content in images of all types, an area of image analysis and image description that has so far not seen the same degree of attention and success as depictions of real-world content (such as people, animals, inanimate objects), and textual content. Diagrammatic image content is defined to a first approximation as line drawings composed of geometrical shapes. The project will look at several subtasks of increasing complexity: (1) locating diagrammatic content in images; (2) breaking down diagrammatic content into component parts; (3) mapping to abstract diagram representations; and (4) generating natural language descriptions of diagrams. In addition to looking at each task separately, we will also look at integrated solutions for two or more of the tasks combined. Neural network methods and hybridised neural network methods will be central to all solutions. Determining what kind of description is appropriate in what kind of context (such as type of user, desired level of abstraction or size of diagram) will be very much part of the research. In summary, the main objectives of the PhD are:1. to develop neural-network-based methods for generating abstract representations of diagram structure and natural language descriptions of diagrammatic content in images;2. to create datasets of paired images containing diagrams, abstract representations and descriptions, for developing, training, and evaluating methods, and for public release as an important data resource for other researchers;3. to create comprehensive evaluation strategies to facilitate thorough evaluation of the developed methods, both by automatic means and involving BPS participants.