MASK FACE INPAINTING BASED ON IMPROVED GENERATIVE ADVERSARIAL NETWORK
Qingyu Liu
liuq@students.national-u.edu.phNational University (Philippines)
https://orcid.org/0009-0000-9774-808X
Roben A. Juanatas
National University (Philippines)
Abstract
Face recognition technology has been widely used in all aspects of people's lives. However, the accuracy of face recognition is greatly reduced due to the obscuring of objects, such as masks and sunglasses. Wearing masks in public has been a crucial approach to preventing illness, especially since the Covid-19 outbreak. This poses challenges to applications such as face recognition. Therefore, the removal of masks via image inpainting has become a hot topic in the field of computer vision. Deep learning-based image inpainting techniques have taken observable results, but the restored images still have problems such as blurring and inconsistency. To address such problems, this paper proposes an improved inpainting model based on generative adversarial network: the model adds attention mechanisms to the sampling module based on pix2pix network; the residual module is improved by adding convolutional branches. The improved inpainting model can not only effectively restore faces obscured by face masks, but also realize the inpainting of randomly obscured images of human faces. To further validate the generality of the inpainting model, tests are conducted on the datasets of CelebA, Paris Street and Place2, and the experimental results show that both SSIM and PSNR have improved significantly.
Keywords:
face inpainting;, generative adversarial network;, residual network;, attention mechanismReferences
Ding, D., Ram, S., & Rodriguez, J. J. (2019). Image Inpainting Using Nonlocal Texture Matching and Nonlinear Filtering. IEEE Transactions on Image Processing, 28(4), 1705–1719. https://doi.org/10.1109/TIP.2018.2880681
DOI: https://doi.org/10.1109/TIP.2018.2880681
Google Scholar
Goodfellow, I. (2014). NIPS 2014 Tutorial: Generative Adversarial Networks. arXiv. https://doi.org/10.48550/arXiv.1701.00160
Google Scholar
He, L., Qiang, Z., Shao, X., Lin, H., Wang, M., & Dai, F. (2022). Research on High-Resolution Face Image Inpainting Method Based on StyleGAN. Electronics, 11(10), 1620. https://doi.org/10.3390/electronics11101620
DOI: https://doi.org/10.3390/electronics11101620
Google Scholar
Hore, A., & Ziou, D. (2010). Image Quality Metrics: PSNR vs. SSIM. 2010 20th International Conference on Pattern Recognition (pp. 2366–2369). IEEE. https://doi.org/10.1109/ICPR.2010.579
DOI: https://doi.org/10.1109/ICPR.2010.579
Google Scholar
Iizuka, S., Simo-Serra, E., & Ishikawa, H. (2017). Globally and locally consistent image completion. ACM Transactions on Graphics, 36(4), 1–14. https://doi.org/10.1145/3072959.3073659
DOI: https://doi.org/10.1145/3072959.3073659
Google Scholar
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017). Imavolge-to-Image Translation with Conditional Adversarial Networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 5967–5976). IEEE. https://doi.org/10.1109/CVPR.2017.632
DOI: https://doi.org/10.1109/CVPR.2017.632
Google Scholar
Jaderberg, M., Simonyan, K., & Zisserman, A. (2016). Spatial Transformer Networks. arXiv. https://doi.org/10.48550/arXiv.1506.02025
Google Scholar
Jiang, Y., Yang, F., Bian, Z., Lu, C., & Xia, S. (2022). Mask removal: Face inpainting via attributes. Multimedia Tools and Applications, 81(21), 29785–29797. https://doi.org/10.1007/s11042-022-12912-1
DOI: https://doi.org/10.1007/s11042-022-12912-1
Google Scholar
Jia, J., & Tang, Ch.-K. (2003). Image repairing: Robust image synthesis by adaptive ND tensor voting. 2003 IEEE Computer Society Conference on Computer Vision springe and Pattern Recognition, 2003. Proceedings. (pp. I-I). IEEE. https://doi.org/10.1109/CVPR.2003.1211414
DOI: https://doi.org/10.1109/CVPR.2003.1211414
Google Scholar
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., & Shi, W. (2017). Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 105–114). IEEE. https://doi.org/10.1109/CVPR.2017.19
DOI: https://doi.org/10.1109/CVPR.2017.19
Google Scholar
Liu, G., Reda, F. A., Shih, K. J., Wang, T.-C., Tao, A., & Catanzaro, B. (2018). Image Inpainting for Irregular Holes Using Partial Convolutions. In V. Ferrari, M. Hebert, C. Sminchisescu, & Y. Weiss (Eds.), Computer Vision – ECCV 2018 (vol. 11215, pp. 89–105). Springer. https://doi.org/10.1007/978-3-030-01252-6_6
DOI: https://doi.org/10.1007/978-3-030-01252-6_6
Google Scholar
Mou, C., Wang, Q., & Zhang, J. (2022). Deep Generalized Unfolding Networks for Image Restoration. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 17378–17389). IEEE. https://doi.org/10.1109/CVPR52688.2022.01688
DOI: https://doi.org/10.1109/CVPR52688.2022.01688
Google Scholar
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016). Context Encoders: Feature Learning by Inpainting. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2536–2544). IEEE. https://doi.org/10.1109/CVPR.2016.278
DOI: https://doi.org/10.1109/CVPR.2016.278
Google Scholar
Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv. https://doi.org/10.48550/arXiv.1505.04597
DOI: https://doi.org/10.1007/978-3-319-24574-4_28
Google Scholar
Rudin, L. I., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena, 60(1–4), 259–268. https://doi.org/10.1016/0167-2789(92)90242-F.
DOI: https://doi.org/10.1016/0167-2789(92)90242-F
Google Scholar
Sagong, M., Shin, Y., Kim, S., Park, S., & Ko, S. (2019). PEPSI: Fast Image Inpainting With Parallel Decoding Network. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 11352–11360). IEEE. https://doi.org/10.1109/CVPR.2019.01162
DOI: https://doi.org/10.1109/CVPR.2019.01162
Google Scholar
Shao, X., Qiang, Z., Dai, F., He, L., & Lin, H. (2022). Face Image Completion Based on GAN Prior. Electronics, 11(13), 1997. https://doi.org/10.3390/electronics11131997
DOI: https://doi.org/10.3390/electronics11131997
Google Scholar
Simakov, D., Caspi, Y., Shechtman, E., & Irani, M. (2008). Summarizing visual data using bidirectional similarity. 2008 IEEE Conference on Computer Vision and Pattern Recognition(CVPR) (pp. 1–8). IEEE. https://doi.org/10.1109/CVPR.2008.4587842
DOI: https://doi.org/10.1109/CVPR.2008.4587842
Google Scholar
Wang, Y., Tao, X., Qi, X., Shen, X., & Jia, J. (2018). Image Inpainting via Generative Multi-column Convolutional Neural Networks. arXiv. https://doi.org/10.48550/arXiv.1810.08771
Google Scholar
Wu, H., Zhou, J., & Li, Y. (2020). Deep Generative Model for Image Inpainting with Local Binary Pattern Learning and Spatial Attention. arXiv. https://doi.org/10.48550/arXiv.2009.01031
Google Scholar
Xie, C., Liu, S., Li, C., Cheng, M.-M., Zuo, W., Liu, X., Wen, S., & Ding, E. (2019). Image Inpainting With Learnable Bidirectional Attention Maps. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 8857–8866). IEEE. https://doi.org/10.1109/ICCV.2019.00895
DOI: https://doi.org/10.1109/ICCV.2019.00895
Google Scholar
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. S. (2019). Free-Form Image Inpainting With Gated Convolution. arXiv. https://doi.org/10.48550/arXiv.1806.03589
DOI: https://doi.org/10.1109/ICCV.2019.00457
Google Scholar
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. S. (2018). Generative Image Inpainting With Contextual Attention. arXiv. https://doi.org/10.48550/arXiv.1801.07892
DOI: https://doi.org/10.1109/CVPR.2018.00577
Google Scholar
Zeng, Y., Fu, J., Chao, H.. & Guo, B. (2019). Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) ( pp.1486–1494). IEEE. https://doi.org/10.1109/CVPR.2019.00158
DOI: https://doi.org/10.1109/CVPR.2019.00158
Google Scholar
Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., & Fu, Y. (2018). Image Super-Resolution Using Very Deep Residual Channel Attention Networks. In V. Ferrari, M. Hebert, C. Sminchisescu, & Y. Weiss (Eds.), Computer Vision – ECCV 2018 (vol. 11211, pp. 294–310). Springer. https://doi.org/10.1007/978-3-030-01234-2_18
DOI: https://doi.org/10.1007/978-3-030-01234-2_18
Google Scholar
Zheng, C., Cham, T.-J., & Cai, J. (2019). Pluralistic Image Completion. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1438–1447). IEEE. https://doi.org/10.1109/CVPR.2019.00153
DOI: https://doi.org/10.1109/CVPR.2019.00153
Google Scholar
Zhou, Z., Siddiquee, M. M. R., Tajbakhsh, N., & Liang, J. (2018). UNet++: A Nested U-Net Architecture for Medical Image Segmentation. arXiv. http://arxiv.org/abs/1807.10165
DOI: https://doi.org/10.1007/978-3-030-00889-5_1
Google Scholar
Authors
Qingyu Liuliuq@students.national-u.edu.ph
National University Philippines
https://orcid.org/0009-0000-9774-808X
Authors
Roben A. JuanatasNational University Philippines
Statistics
Abstract views: 165PDF downloads: 125
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
All articles published in Applied Computer Science are open-access and distributed under the terms of the Creative Commons Attribution 4.0 International License.
Similar Articles
- Muaayed F. AL-RAWI, CONVENTIONAL ENERGY EFFICIENT ROUTING PROTOCOLS IN WIRELESS SENSOR NETWORKS , Applied Computer Science: Vol. 16 No. 3 (2020)
- Md. Torikur RAHMAN, Mohammad ALAUDDIN, Uttam Kumar DEY, Dr. A.H.M. Saifullah SADI, ADAPTIVE SECURE AND EFFICIENT ROUTING PROTOCOL FOR ENHANCE THE PERFORMANCE OF MOBILE AD HOC NETWORK , Applied Computer Science: Vol. 19 No. 3 (2023)
- Md. Torikur RAHMAN, A NOVEL APPROACH TO ENHANCE THE PERFORMANCE OF MOBILE AD HOC NETWORK (MANET) THROUGH A NEW BANDWIDTH OPTIMIZATION TECHNIQUE , Applied Computer Science: Vol. 15 No. 2 (2019)
- Nataliya SHABLIY, Serhii LUPENKO, Nadiia LUTSYK, Oleh YASNIY, Olha MALYSHEVSKA, KEYSTROKE DYNAMICS ANALYSIS USING MACHINE LEARNING METHODS , Applied Computer Science: Vol. 17 No. 4 (2021)
- Sunil Kumar B L, Sharmila Kumari M, RGB-D FACE RECOGNITION USING LBP-DCT ALGORITHM , Applied Computer Science: Vol. 17 No. 3 (2021)
- Puppala Praneeth, Majety Sathvika, Vivek Kommareddy, Madala Sarath, Saran Mallela, Koneru Suvarna Vani, Prasun Chkrabarti, CLASSIFICATION OF PARKINSON'S DISEASE IN BRAIN MRI IMAGES USING DEEP RESIDUAL CONVOLUTIONAL NEURAL NETWORK , Applied Computer Science: Vol. 19 No. 2 (2023)
- Edyta ŁUKASIK, Wiktor FLIS, EFFICIENCY COMPARISON OF NETWORKS IN HANDWRITTEN LATIN CHARACTERS RECOGNITION WITH DIACRITICS , Applied Computer Science: Vol. 19 No. 4 (2023)
- Robert KARPIŃSKI, Jakub GAJEWSKI, Jakub SZABELSKI, Dalibor BARTA, APPLICATION OF NEURAL NETWORKS IN PREDICTION OF TENSILE STRENGTH OF ABSORBABLE SUTURES , Applied Computer Science: Vol. 13 No. 4 (2017)
- Manikandan SRIDHARAN, Delphin Carolina RANI ARULANANDAM, Rajeswari K CHINNASAMY, Suma THIMMANNA, Sivabalaselvamani DHANDAPANI, RECOGNITION OF FONT AND TAMIL LETTER IN IMAGES USING DEEP LEARNING , Applied Computer Science: Vol. 17 No. 2 (2021)
- Wulan Dewi, Wiranto Herry Utomo, PLANT CLASSIFICATION BASED ON LEAF EDGES AND LEAF MORPHOLOGICAL VEINS USING WAVELET CONVOLUTIONAL NEURAL NETWORK , Applied Computer Science: Vol. 17 No. 1 (2021)
You may also start an advanced similarity search for this article.
