MASK FACE INPAINTING BASED ON IMPROVED GENERATIVE ADVERSARIAL NETWORK
Qingyu Liu
liuq@students.national-u.edu.phNational University (Philippines)
https://orcid.org/0009-0000-9774-808X
Roben A. Juanatas
National University (Philippines)
Abstract
Face recognition technology has been widely used in all aspects of people's lives. However, the accuracy of face recognition is greatly reduced due to the obscuring of objects, such as masks and sunglasses. Wearing masks in public has been a crucial approach to preventing illness, especially since the Covid-19 outbreak. This poses challenges to applications such as face recognition. Therefore, the removal of masks via image inpainting has become a hot topic in the field of computer vision. Deep learning-based image inpainting techniques have taken observable results, but the restored images still have problems such as blurring and inconsistency. To address such problems, this paper proposes an improved inpainting model based on generative adversarial network: the model adds attention mechanisms to the sampling module based on pix2pix network; the residual module is improved by adding convolutional branches. The improved inpainting model can not only effectively restore faces obscured by face masks, but also realize the inpainting of randomly obscured images of human faces. To further validate the generality of the inpainting model, tests are conducted on the datasets of CelebA, Paris Street and Place2, and the experimental results show that both SSIM and PSNR have improved significantly.
Keywords:
face inpainting;, generative adversarial network;, residual network;, attention mechanismReferences
Ding, D., Ram, S., & Rodriguez, J. J. (2019). Image Inpainting Using Nonlocal Texture Matching and Nonlinear Filtering. IEEE Transactions on Image Processing, 28(4), 1705–1719. https://doi.org/10.1109/TIP.2018.2880681
DOI: https://doi.org/10.1109/TIP.2018.2880681
Google Scholar
Goodfellow, I. (2014). NIPS 2014 Tutorial: Generative Adversarial Networks. arXiv. https://doi.org/10.48550/arXiv.1701.00160
Google Scholar
He, L., Qiang, Z., Shao, X., Lin, H., Wang, M., & Dai, F. (2022). Research on High-Resolution Face Image Inpainting Method Based on StyleGAN. Electronics, 11(10), 1620. https://doi.org/10.3390/electronics11101620
DOI: https://doi.org/10.3390/electronics11101620
Google Scholar
Hore, A., & Ziou, D. (2010). Image Quality Metrics: PSNR vs. SSIM. 2010 20th International Conference on Pattern Recognition (pp. 2366–2369). IEEE. https://doi.org/10.1109/ICPR.2010.579
DOI: https://doi.org/10.1109/ICPR.2010.579
Google Scholar
Iizuka, S., Simo-Serra, E., & Ishikawa, H. (2017). Globally and locally consistent image completion. ACM Transactions on Graphics, 36(4), 1–14. https://doi.org/10.1145/3072959.3073659
DOI: https://doi.org/10.1145/3072959.3073659
Google Scholar
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017). Imavolge-to-Image Translation with Conditional Adversarial Networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 5967–5976). IEEE. https://doi.org/10.1109/CVPR.2017.632
DOI: https://doi.org/10.1109/CVPR.2017.632
Google Scholar
Jaderberg, M., Simonyan, K., & Zisserman, A. (2016). Spatial Transformer Networks. arXiv. https://doi.org/10.48550/arXiv.1506.02025
Google Scholar
Jiang, Y., Yang, F., Bian, Z., Lu, C., & Xia, S. (2022). Mask removal: Face inpainting via attributes. Multimedia Tools and Applications, 81(21), 29785–29797. https://doi.org/10.1007/s11042-022-12912-1
DOI: https://doi.org/10.1007/s11042-022-12912-1
Google Scholar
Jia, J., & Tang, Ch.-K. (2003). Image repairing: Robust image synthesis by adaptive ND tensor voting. 2003 IEEE Computer Society Conference on Computer Vision springe and Pattern Recognition, 2003. Proceedings. (pp. I-I). IEEE. https://doi.org/10.1109/CVPR.2003.1211414
DOI: https://doi.org/10.1109/CVPR.2003.1211414
Google Scholar
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., & Shi, W. (2017). Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 105–114). IEEE. https://doi.org/10.1109/CVPR.2017.19
DOI: https://doi.org/10.1109/CVPR.2017.19
Google Scholar
Liu, G., Reda, F. A., Shih, K. J., Wang, T.-C., Tao, A., & Catanzaro, B. (2018). Image Inpainting for Irregular Holes Using Partial Convolutions. In V. Ferrari, M. Hebert, C. Sminchisescu, & Y. Weiss (Eds.), Computer Vision – ECCV 2018 (vol. 11215, pp. 89–105). Springer. https://doi.org/10.1007/978-3-030-01252-6_6
DOI: https://doi.org/10.1007/978-3-030-01252-6_6
Google Scholar
Mou, C., Wang, Q., & Zhang, J. (2022). Deep Generalized Unfolding Networks for Image Restoration. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 17378–17389). IEEE. https://doi.org/10.1109/CVPR52688.2022.01688
DOI: https://doi.org/10.1109/CVPR52688.2022.01688
Google Scholar
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016). Context Encoders: Feature Learning by Inpainting. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2536–2544). IEEE. https://doi.org/10.1109/CVPR.2016.278
DOI: https://doi.org/10.1109/CVPR.2016.278
Google Scholar
Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv. https://doi.org/10.48550/arXiv.1505.04597
DOI: https://doi.org/10.1007/978-3-319-24574-4_28
Google Scholar
Rudin, L. I., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena, 60(1–4), 259–268. https://doi.org/10.1016/0167-2789(92)90242-F.
DOI: https://doi.org/10.1016/0167-2789(92)90242-F
Google Scholar
Sagong, M., Shin, Y., Kim, S., Park, S., & Ko, S. (2019). PEPSI: Fast Image Inpainting With Parallel Decoding Network. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 11352–11360). IEEE. https://doi.org/10.1109/CVPR.2019.01162
DOI: https://doi.org/10.1109/CVPR.2019.01162
Google Scholar
Shao, X., Qiang, Z., Dai, F., He, L., & Lin, H. (2022). Face Image Completion Based on GAN Prior. Electronics, 11(13), 1997. https://doi.org/10.3390/electronics11131997
DOI: https://doi.org/10.3390/electronics11131997
Google Scholar
Simakov, D., Caspi, Y., Shechtman, E., & Irani, M. (2008). Summarizing visual data using bidirectional similarity. 2008 IEEE Conference on Computer Vision and Pattern Recognition(CVPR) (pp. 1–8). IEEE. https://doi.org/10.1109/CVPR.2008.4587842
DOI: https://doi.org/10.1109/CVPR.2008.4587842
Google Scholar
Wang, Y., Tao, X., Qi, X., Shen, X., & Jia, J. (2018). Image Inpainting via Generative Multi-column Convolutional Neural Networks. arXiv. https://doi.org/10.48550/arXiv.1810.08771
Google Scholar
Wu, H., Zhou, J., & Li, Y. (2020). Deep Generative Model for Image Inpainting with Local Binary Pattern Learning and Spatial Attention. arXiv. https://doi.org/10.48550/arXiv.2009.01031
Google Scholar
Xie, C., Liu, S., Li, C., Cheng, M.-M., Zuo, W., Liu, X., Wen, S., & Ding, E. (2019). Image Inpainting With Learnable Bidirectional Attention Maps. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 8857–8866). IEEE. https://doi.org/10.1109/ICCV.2019.00895
DOI: https://doi.org/10.1109/ICCV.2019.00895
Google Scholar
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. S. (2019). Free-Form Image Inpainting With Gated Convolution. arXiv. https://doi.org/10.48550/arXiv.1806.03589
DOI: https://doi.org/10.1109/ICCV.2019.00457
Google Scholar
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. S. (2018). Generative Image Inpainting With Contextual Attention. arXiv. https://doi.org/10.48550/arXiv.1801.07892
DOI: https://doi.org/10.1109/CVPR.2018.00577
Google Scholar
Zeng, Y., Fu, J., Chao, H.. & Guo, B. (2019). Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) ( pp.1486–1494). IEEE. https://doi.org/10.1109/CVPR.2019.00158
DOI: https://doi.org/10.1109/CVPR.2019.00158
Google Scholar
Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., & Fu, Y. (2018). Image Super-Resolution Using Very Deep Residual Channel Attention Networks. In V. Ferrari, M. Hebert, C. Sminchisescu, & Y. Weiss (Eds.), Computer Vision – ECCV 2018 (vol. 11211, pp. 294–310). Springer. https://doi.org/10.1007/978-3-030-01234-2_18
DOI: https://doi.org/10.1007/978-3-030-01234-2_18
Google Scholar
Zheng, C., Cham, T.-J., & Cai, J. (2019). Pluralistic Image Completion. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1438–1447). IEEE. https://doi.org/10.1109/CVPR.2019.00153
DOI: https://doi.org/10.1109/CVPR.2019.00153
Google Scholar
Zhou, Z., Siddiquee, M. M. R., Tajbakhsh, N., & Liang, J. (2018). UNet++: A Nested U-Net Architecture for Medical Image Segmentation. arXiv. http://arxiv.org/abs/1807.10165
DOI: https://doi.org/10.1007/978-3-030-00889-5_1
Google Scholar
Authors
Qingyu Liuliuq@students.national-u.edu.ph
National University Philippines
https://orcid.org/0009-0000-9774-808X
Authors
Roben A. JuanatasNational University Philippines
Statistics
Abstract views: 316PDF downloads: 193
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
All articles published in Applied Computer Science are open-access and distributed under the terms of the Creative Commons Attribution 4.0 International License.
Similar Articles
- Rowell HERNANDEZ, Robert ATIENZA, CAREER TRACK PREDICTION USING DEEP LEARNING MODEL BASED ON DISCRETE SERIES OF QUANTITATIVE CLASSIFICATION , Applied Computer Science: Vol. 17 No. 4 (2021)
- Anna MACHROWSKA, Robert KARPIŃSKI, Józef JONAK, Jakub SZABELSKI, NUMERICAL PREDICTION OF THE COMPONENT-RATIO-DEPENDENT COMPRESSIVE STRENGTH OF BONE CEMENT , Applied Computer Science: Vol. 16 No. 3 (2020)
- Grzegorz RADZKI, Amila THIBBOTUWAWA, Grzegorz BOCEWICZ, UAVS FLIGHT ROUTES OPTIMIZATION IN CHANGING WEATHER CONDITIONS – CONSTRAINT PROGRAMMING APPROACH , Applied Computer Science: Vol. 15 No. 3 (2019)
- Michał TOMCZYK, Anna PLICHTA, Mariusz MIKULSKI, APPLICATION OF WAVELET – NEURAL METHOD TO DETECT BACKLASH ZONE IN ELECTROMECHANICAL SYSTEMS GENERATING NOISES , Applied Computer Science: Vol. 15 No. 4 (2019)
- Saleh ALBAHLI, A DEEP ENSEMBLE LEARNING METHOD FOR EFFORT-AWARE JUST-IN-TIME DEFECT PREDICTION , Applied Computer Science: Vol. 16 No. 3 (2020)
- Krzysztof OSTROWSKI, AN EFFECTIVE METAHEURISTIC FOR TOURIST TRIP PLANNING IN PUBLIC TRANSPORT NETWORKS , Applied Computer Science: Vol. 14 No. 2 (2018)
- Marcin TOMCZYK, Barbara BOROWIK, Bohdan BOROWIK, IDENTIFICATION OF THE MASS INERTIA MOMENT IN AN ELECTROMECHANICAL SYSTEM BASED ON WAVELET–NEURAL METHOD , Applied Computer Science: Vol. 14 No. 2 (2018)
- Maciej NABOŻNY, ASYNCHRONOUS INFORMATION DISTRIBUTION AND CLUSTER STATE SYNCHRONIZATION , Applied Computer Science: Vol. 14 No. 1 (2018)
- Muayed S AL-HUSEINY, Ahmed S SAJIT, BREAST CANCER CAD SYSTEM BY USING TRANSFER LEARNING AND ENHANCED ROI , Applied Computer Science: Vol. 18 No. 1 (2022)
- Mahmoud BAKR, Sayed ABDEL-GABER, Mona NASR, Maryam HAZMAN, TOMATO DISEASE DETECTION MODEL BASED ON DENSENET AND TRANSFER LEARNING , Applied Computer Science: Vol. 18 No. 2 (2022)
You may also start an advanced similarity search for this article.
