Collaborative Research: RI: Small: Unsupervised Islamicate Manuscript Transcription via Lacunae Reconstruction

该奖项涉及伊斯兰手稿的手写文本识别(HTR，自动将手写手稿图像转录成符号文本的任务)，这一领域涵盖了起源于前现代伊斯兰世界(7-19世纪)的波斯和阿拉伯书面传统。现代文本的HTR本身就是一个具有挑战性的问题，受到了机器学习(ML)和人工智能(AI)领域的极大关注。然而，现代文本在HTR研究中的主导地位在某种程度上正在减弱：目前的技术基于现代数据相对稳健，当代书面媒体制作几乎已经完全数字化。相比之下，历史手稿相对较少受到ML和AI的关注，同时也代表着ML技术产生影响的特殊机会和一系列独特的挑战。具体地说，伊斯兰世界的书面传统共同构成了前现代世界最大的--如果不是最大的--人类文化生产档案之一。过去十年的扫描和数字化努力使大量收藏的伊斯兰手稿的图像向公众开放。然而，对于大多数学术用途来说，这些数据仍然是“锁定的”，因为它还没有被转录成许多类型的分析所需的符号文本。事实上，伊斯兰手稿中使用的手稿风格差异如此之大，与现代形式差异如此之大，以至于即使是手动仔细阅读这些文本也需要专家培训，因此仅限于一小部分研究人员。这个项目的主要成果将是新的技术，通过准确地转录来‘解锁’伊斯兰文字传统。因此，该项目有可能改变伊斯兰和近东研究等人文学科，使图书馆能够准确地转录整个藏书，并进一步允许个别研究人员准确地转录西方正典之外的手稿。最后，这项研究还将支持加州大学圣地亚哥分校和马里兰大学不同学科的研究生的跨学科培训。目前的HTR技术需要大量领域内监督的培训数据，以产生高精度的转录。这些现代方法背后的神经结构在某种程度上能够在更大和更多样化的转录数据集合上进行训练时，在现代笔迹风格中推广。然而，由于它们的局限性，这些技术对于伊斯兰文本的大规模转录来说是不切实际的，原因有两个：(1)伊斯兰手稿中的抄写手写变化比现代手写中的风格变化要明显得多；(2)可以用作监督训练数据的伊斯兰手稿的转录极其稀缺，因为准确的手动转录需要专家培训。该项目将为伊斯兰HTR开发一个新的无监督学习框架，中心是一项新的预培训任务：腔隙重建。这种新的方法通过学习重建未标记的手稿图像的掩蔽区域--即腔隙--来训练手稿文本行图像的神经编码器。这一完全无监督的培训标准隐含地激励模型发现并编码离散。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This award tackles handwritten text recognition (HTR, the task of automatically transcribing images of handwritten manuscripts into symbolic text) for Islamicate manuscripts, a domain that encompasses Persian and Arabic written traditions originating in the premodern Islamic world (7th-19th centuries). HTR for modern text is itself a challenging problem that has received substantial attention from the fields of machine learning (ML) and artificial intelligence (AI). However, the predominance of modern text in HTR research is, to some extent, waning: current techniques are relatively robust on modern data, and contemporary written media production is already almost entirely digital. In contrast, historical manuscripts have received comparatively less attention from ML and AI, and at the same time represent both an exceptional opportunity for impact and a set of unique challenges for ML techniques. Specifically, the written traditions of the Islamicate world together form one of the largest -- if not the largest -- archives of human cultural production of the premodern world. Scanning and digitization efforts over the last decade have made images of Islamicate manuscripts in a large number of collections available to the public. However, this data remains ‘locked’ for most scholarly uses because it has not been transcribed into symbolic text which is required for many types of analysis. In fact, the script styles used in Islamicate manuscripts -- 'scribal hands' -- vary so widely and differ so substantially from modern forms that even manual close reading of these texts requires expert training and is thus limited to a small subset of researchers. The primary outcome of this project will be new techniques that 'unlock' the Islamicate written tradition by accurately transcribing it. As a result, this project has the potential to be transformative for humanities disciplines such as Islamic and Near Eastern Studies by enabling libraries to accurately transcribe entire collections and, further, by allowing individual researchers to accurately transcribe manuscripts outside the western canon. Finally, this research will also support interdisciplinary training of a diverse set of graduate students at the University of California San Diego and the University of Maryland.Current techniques for HTR require large amounts of in-domain supervised training data in order to produce highly accurate transcriptions. The neural architectures behind these modern methods are capable of generalizing, to some degree, across modern handwriting styles when trained on larger and more diverse collections of transcribed data. However, their limitations make these techniques impractical for large-scale transcription of Islamicate texts for two reasons: (1) scribal hand variation across Islamicate manuscripts is much more pronounced than stylistic variation in modern handwriting; and (2) transcriptions of Islamicate manuscripts that can be used as supervised training data are extremely scarce because accurate manual transcription requires expert training. This project will develop a new unsupervised learning framework for Islamicate HTR centered around a novel pretraining task: lacuna reconstruction. The new approach trains a neural encoder for images of manuscript text lines by learning to reconstruct masked regions -- i.e. lacaunae -- of unlabeled manuscript images. This completely unsupervised training criterion implicitly incentivizes the model to discover and encode discreteThis award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.