Combining Image and Graph-based Neural Networks for Handwriting Recognition

该提案旨在为文档分析任务（如分类或检索）开发模型，数据合成和训练方法，允许应用深度学习模型，而无需手动创建注释。特别是，考虑到历史文献或低资源语言，对标记数据的大量需求阻碍了基于学习的方法的应用。在提议的项目的一个关键因素是利用笔迹的结构性质。除了视觉外观之外，底层结构还可以用图形表示。几何深度学习的最新发展允许将这种结构元素集成到模型设计层面。此外，显式地对几何组件进行建模作为一种正则化形式，增加了开发模型的适应性和泛化能力。然后在尽可能少的监督下进行结合视觉和几何信息的模型训练。因此，将开发包括结构表示在内的数据综合方法。该项目通过开发显式模拟手写几何成分的方法，扩展了基于深度学习的文档分析的最新技术。由此减少的训练数据需求开辟了一系列不同的应用领域和任务。

This proposal aims to develop models, data synthesis and training methods for document analysis tasks such as classification or retrieval that allow the application of deep learning models without the need for manually created annotations. Especially, considering historic documents or low resource language the heavy demand for labeled data hinders the application of learning-based methodology. A key factor in the proposed project is the exploitation of the structural nature of handwriting. In Addition to the visual appearance, the underlying structure can represented as a graph. Recent developments in geometric deep learning allow to integrate this structural element on the level of model design. Additionally, explicitly modeling the geometric component serves as a form of regularization, increasing adaptability and the generalization capabilities of the developed models. Training of a model combining visual and geometrical information is then performed with as little supervision as possible. Therefore, data synthesis approaches will be developed that also include structural representations. This project extends the state-of-the art in deep learning based document analysis by developing methods that explicitly model the geometric components of handwriting. The resulting reduction of training data demand opens up a diverse set of application areas and tasks.