Thanka Mural Inpainting Based on Multi-Scale Adaptive Partial Convolution and Stroke-Like Mask
Thanka Mural Inpainting Based on Multi-Scale Adaptive Partial Convolution and Stroke-Like Mask
复制标题
基于多尺度自适应部分卷积和类描边掩模的Thanka壁画修复
DOI：
10.1109/tip.2021.3064268
复制
发表时间：
2021-01-01
影响因子：
10.6
通讯作者：
Li, Shuo
中科院分区：
文献类型：
--
作者：
Wang, Nianyi;Wang, Weilan;Li, Shuo
唐卡壁画是西藏重要的文化遗产，但许多珍贵的壁画在历史上遭到破坏。唐卡壁画的修复对于西藏文化遗产的保护具有重要意义。部分卷积算法在修复不规则孔洞方面表现出色，在唐卡壁画修复中具有很大的应用潜力。然而，现有的基于部分卷积的方法在解决Thanka修复问题时面临三个挑战：1）单尺度部分卷积无法正确提取Thanka壁画中多尺度对象的特征; 2）现有的矩形或任意掩模无法有效模拟和学习类笔画Thanka修复模式;三是破损的唐卡壁画无法恢复原有内容。针对这些问题，提出了一种基于多尺度自适应部分卷积和类笔画掩模的Thanka壁画修复方法。该方法由三部分组成：1）核级多尺度自适应部分卷积（MAPConv），用于准确区分有效像素和无效像素，并提取多尺度对象的特征; 2）参数可配置的类笔画掩模生成方法，用于模拟和学习类笔画Thanka修复模式; 3）基于MAPConv Unet和不同损失函数的两阶段学习框架，用于恢复唐卡壁画的原始内容。通过对唐卡壁画的模拟和真实的损伤的实验表明，该方法在小数据集（N=2780）上运行良好，生成了逼真的壁画内容，并以较高的速度恢复了受损的唐卡壁画（在$512\times512 $的图像中，多个孔洞的恢复速度为600 ms）。所提出的端到端的方法可以应用到其他小的基于小块的修复任务。
Thanka murals are important cultural heritages of Tibet, but many precious murals were damaged during history. Thanka mural restoration is very important for the protection of Tibetan cultural heritage. Partial convolution has great potential for Thanka mural restoration due to its outstanding performance for inpainting irregular holes. However, three challenges prevent the existing partial convolution-based methods from solving Thanka restoration problems: 1) the features of multi-scale objects in Thanka murals cannot be extracted correctly because of single-scale partial convolution; 2) the stroke-like Thanka inpainting mode cannot be effectively simulated and learned by existing rectangular or arbitrary masks; and 3) the original content of damaged Thanka murals cannot be restored. To resolve these problems, we propose a Thanka mural inpainting method based on multi-scale adaptive partial convolution and stroke-like masks. The proposed method consists of three parts: 1) a kernel-level multi-scale adaptive partial convolution (MAPConv) to accurately discriminate valid pixels from invalid pixels, and to extract the features of multi-scale objects; 2) a parameter-configurable stroke-like mask generation method to simulate and learn the stroke-like Thanka inpainting mode; and 3) a 2-phase learning framework based on MAPConv Unet and different loss functions to restore the original content of Thanka murals. Experiments on both simulated and real damages of Thanka murals demonstrated that our approach works well on a small dataset (N=2780), generates realistic mural content, and restores the damaged Thanka murals with high speed (600 ms for multiple holes in $512\times 512$ images). The proposed end-to-end method can be applied to other small datasets-based inpainting tasks.