Recognition and Description of Diagrammatic Content in Images Using Neural Networks

这个博士项目的目的是在所有类型的图像，图像分析和图像描述的一个领域，到目前为止还没有看到相同程度的关注和成功的真实世界的内容（如人，动物，无生命的物体）和文本内容的解释。图解图像内容被定义为由几何形状组成的线条画的第一近似。该项目将着眼于几个日益复杂的子任务：（1）在图像中定位图表内容;（2）将图表内容分解为组件;（3）映射到抽象图表表示;（4）生成图表的自然语言描述。除了单独查看每个任务外，我们还将查看两个或多个任务的集成解决方案。神经网络方法和混合神经网络方法将成为所有解决方案的核心。确定什么样的描述适合什么样的上下文（例如用户类型，期望的抽象级别或图的大小）将是研究的重要组成部分。总之，博士的主要目标是：1。开发基于神经网络的方法，用于生成图形结构的抽象表示和图像中图形内容的自然语言描述;2.创建包含图表、抽象表示和描述的成对图像的数据集，用于开发、训练和评估方法，并作为其他研究人员的重要数据资源公开发布;3.制定全面的评估战略，以促进通过自动手段和BPS参与者对开发的方法进行全面评估。

This PhD project is aimed at diagrammatic content in images of all types, an area of image analysis and image description that has so far not seen the same degree of attention and success as depictions of real-world content (such as people, animals, inanimate objects), and textual content. Diagrammatic image content is defined to a first approximation as line drawings composed of geometrical shapes. The project will look at several subtasks of increasing complexity: (1) locating diagrammatic content in images; (2) breaking down diagrammatic content into component parts; (3) mapping to abstract diagram representations; and (4) generating natural language descriptions of diagrams. In addition to looking at each task separately, we will also look at integrated solutions for two or more of the tasks combined. Neural network methods and hybridised neural network methods will be central to all solutions. Determining what kind of description is appropriate in what kind of context (such as type of user, desired level of abstraction or size of diagram) will be very much part of the research. In summary, the main objectives of the PhD are:1. to develop neural-network-based methods for generating abstract representations of diagram structure and natural language descriptions of diagrammatic content in images;2. to create datasets of paired images containing diagrams, abstract representations and descriptions, for developing, training, and evaluating methods, and for public release as an important data resource for other researchers;3. to create comprehensive evaluation strategies to facilitate thorough evaluation of the developed methods, both by automatic means and involving BPS participants.