DFIE3D: 3D-Aware Disentangled Face Inversion and Editing via Facial-Contrastive Learning

Authors: Zhu, X., Zhou, J., You, L., Yang, X., Chang, J., Zhang, J.J. and Zeng, D.

Journal: IEEE Transactions on Circuits and Systems for Video Technology

Volume: 34

Issue: 9

Pages: 8310-8326

eISSN: 1558-2205

ISSN: 1051-8215

DOI: 10.1109/TCSVT.2024.3377121

Abstract:

Recent advances in NeRF-based 3D-aware GANs have achieved outstanding performance, especially in the realm of human facial representations, making projection of facial images back into their latent space superior and preferable compared to 2D GAN inversion. However, the direct application of 2DGAN inversion techniques to 3DGAN raises challenges due to potential appearance distortions and geometric inconsistences. To tackle these issues, this work presents a novel integrated framework that combines a composite inversion pipeline in both the SS and W+ spaces and integrates a contrastive-based training strategy, ensuring proficient disentanglement within the module. Moreover, we design a facial semantic manipulation technique based on dimensional analysis of the latent code, which is fully compatible with the proposed 3DGAN inversion pipeline. Comprehensive experimental validations substantiate the effectiveness of the proposed approach in executing 3d-aware face inversion and semantic editing tasks, presenting a robust technological solution for a diverse array of digital human modeling applications in the downstream.

https://eprints.bournemouth.ac.uk/40106/

Source: Scopus