Research on attention co-modulated adversarial network for image completion
Ancient murals,as a precious cultural heritage,have important historical,cultural,artistic,and scientific values.However,due to natural and human factors,they have suffered varying degrees of damage,including missing parts,pigment shedding,and other issues.These problems seriously affect the conservation and utilization of the murals.Conventional manual repair or duplications are performed on some ancient murals,which may cause secondary damages.Besides,manual repair has some drawbacks,such as being time-consuming,laborious,and inefficient,making it inapplicable for large-scale mural restoration.In contrast,virtual repair methods recover missing information in digital images of ancient murals without damaging the mural bodies,providing a potentially attractive non-contact solution for mural conservation.Although existing virtual repair methods have achieved remarkable results,they assume that the locations of damaged areas are known beforehand.In practice,however,it is difficult to accurately locate the damaged areas.Moreover,for damaged areas with complex structures,these methods may cause semantic errors and serious artifacts due to low modeling efficiency.To address these issues,we propose a novel attention co-modulated adversarial network for blind mural image completion.It automatically detects and repairs damaged areas in mural images,in which both multi-attention modules and gated convolutions are built to extract location masks and fill missing holes respectively.Our network consists of two stages:damage detection and hole inpainting.In the first stage,a multi-path attention module is designed and integrated with both region proposal network and branch prediction modules to estimate the location mask of damaged areas.In the second stage,with the estimated location mask available,the co-modulated generative adversarial network module incorporates with gated convolution to repair the damaged areas.More specifically,the damage detection stage includes two parts:coarse segmentation and fine segmentation.In the coarse segmentation,given a damaged mural image,the features extracted from the convolutional layer of the input damaged image are first divided into G cardinality groups to control the connection between the input and output.Each cardinality group is further divided into R subgroups to learn the features of different channels.In the fine segmentation,the feature pyramid structure is first used to fuse the extracted features by using a 1 × 1 convolution.Then the fused features are upsampled and added to the previous features by residual learning to increase the accuracy of damage detection.Moreover,a gated convolution is introduced to build the collaborative modulation-based generative adversarial network module,making the location mask into a learnable soft gating map.We created a custom mural dataset by collecting 5 500 ancient mural images mainly from the Tang Dynasty,with 4,400 images for training,550 images for validation,and 550 images for testing.We compared our model with four state-of-the-art deep learning models on ancient mural image datasets.Our model increases the average precision(AP)value by 15％compared to Mask R-CNN and reduces the Fréchet Inception Distance(FID)score by 4.6％compared to CoModGAN.Comparative experiments demonstrates our model improves the results of mural image completion and generally outperforms the existing popular methods in qualitative and quantitative evaluations.We propose a blind mural image completion model consisting of damage detection and hole inpainting.We introduce multi-path attention and gated convolution into damage detection and hole inpainting respectively.The model first employs the multi-path attention module to extract rich semantic features for rough segmentation of damaged areas,then adopts the region proposal network and prediction branch for fine segmentation,and finally employs the collaborative modulation-based generative adversarial network module via gated convolution to fill the missing areas for constructing a complete repaired mural image.The ablation study confirms the effectiveness of the multi-path attention module and gated convolution in mural image completion.Our experimental results show our method automatically identifies the precise location mask of damaged areas in mural images,produces visually realistic completed mural images.It outperforms baseline methods in both qualitative and quantitative evaluations.
国家科技期刊平台
NSTL
万方数据
