Deep learning for the extraction of sketches from spectral images of historical paintings
Deep learning for the extraction of sketches from spectral images of historical paintings
复制标题
DOI：
10.1117/12.2593680
复制
发表时间：
2021-06
影响因子：
3.4
通讯作者：
Qunxi Zhang;Shanshan Cui;Lu Liu;Jiaxin Wang;Jun Wang;Erlei Zhang;Jinye Peng;S. Kogou;Florence S. Liggins;Haida Liang
中科院分区：
文献类型：
--
作者：
Qunxi Zhang;Shanshan Cui;Lu Liu;Jiaxin Wang;Jun Wang;Erlei Zhang;Jinye Peng;S. Kogou;Florence S. Liggins;Haida Liang
绘画文物的素描可以是绘画风格的最具代表性的。素描的提取是一个不可或缺的过程，被保护者和艺术历史学家用于记录，并让艺术家通过复制和绘画来学习历史绘画风格。但目前草图提取主要是手工绘制，不仅耗时，而且主观性强，依赖经验。因此，准确性和效率都需要提高。近年来，随着机器学习的发展，出现了一系列基于边缘检测的图像提取方法。然而，现有的大多数方法只能在草图保存良好的情况下才能成功提取，而对于草图褪色或保存问题严重的数据，提取方法需要改进。在抑制退化区域和重叠绘制的效果的同时，提取突出草图的条带是有益的。提出了一种基于高光谱图像和深度学习的草图提取方法。首先，收集高光谱图像数据，并通过草图的先验知识提取对草图敏感的波段（例如，如果草图由碳墨水制成，则将选择近红外波段），并且使用包括大量现有自然图像的数据集来预训练双向级联网络（BDCN）。然后利用专家绘制的彩绘文物图像对模型中的网络参数进行微调，以解决彩绘文物草图数据集不足的问题，增强模型的泛化能力。最后，使用U-net网络进一步抑制噪声，即不需要的信息，使草图更清晰。实验结果表明，该方法不仅可以有效地从理想数据中提取草图，而且可以从草图褪色甚至有噪声干扰的数据中提取清晰的草图。与其他六种先进的基于边缘检测的方法相比，该方法在直观、客观上具有上级优势，具有良好的应用前景。还将所提出的深度学习方法与使用自组织映射（SOM）的无监督聚类方法进行了比较，SOM是一种“浅层学习”方法，其中相似光谱的像素被分组为聚类，而无需专家进行数据标记。
The sketches of painted cultural objects can be the most indicative of the style of paintings. Extraction of the sketches is an integral process used by conservators and art historians for documentation and for artists to learn historical painting styles through copying and painting. However, at present, sketch extraction is mainly manually drawn, which is not only time-consuming, but also subjective and dependent on experience. Therefore, both accuracy and efficiency need to be improved. In recent years, with the development of machine learning, a series of extraction methods based on edge detection have emerged. However, most of the existing methods can only perform successful extraction if the sketches are well preserved , but for the data with faded sketches or severe conservation issues, the extraction methods need to be improved. It is beneficial to extract the bands that accentuate the sketches while suppressing the effects of the degraded areas and the overlapping paints. We propose a sketch extraction method based on hyperspectral image and deep learning. Firstly, the hyperspectral image data is collected and the bands sensitive to the sketches are extracted by a prior knowledge of the sketches (e.g. near infrared bands will be chosen if the sketches are made of carbon ink), and a dataset including a large number of existing natural images is used to pre-train the bi-directional cascade network (BDCN). The network parameters in the model are then fine-tuned by using the images of painted cultural objects drawn by experts, so as to solve the problem of insufficient sketch dataset of painted cultural objects and enhance the generalization ability of the model. Finally, the U-net network is used to further suppress the noise, i.e. unwanted information, and make the sketch clearer. The experimental results show that the proposed method can not only effectively extract sketch from ideal data, but also extract clear sketches from data with faded sketches and even with noise interference. It is superior to the other six advanced based on edge detection methods in visual and objective comparison, and has a good application prospect. The proposed deep learning method is also compared with an unsupervised clustering method using Self-Organising Map (SOM) which is a ‘shallow learning’ method where pixels of similar spectra are grouped into clusters without the need for data labeling by experts.