REALY: Rethinking the Evaluation of 3D Face Reconstruction

ECCV 2022

Zenghao Chai*, Haoxian Zhang*, Jing Ren, Di Kang,

Zhengzhuo Xu, Xuefei Zhe, Chun Yuan†, Linchao Bao†

(* Equal Contribution, † Corresponding Author)

Abstract

The evaluation of 3D face reconstruction results typically relies on a rigid shape alignment between the estimated 3D model and the ground-truth scan. We observe that aligning two shapes with different reference points can largely affect the evaluation results. This poses difficulties for precisely diagnosing and improving a 3D face reconstruction method. In this paper, we propose a novel evaluation approach with a new benchmark REALY, consists of 100 globally aligned face scans with accurate facial keypoints, high-quality region masks, and topology-consistent meshes. Our approach performs region-wise shape alignment and leads to more accurate, bidirectional correspondences during computing the shape errors. The fine-grained, region-wise evaluation results provide us detailed understandings about the performance of state-of-the-art 3D face reconstruction methods. For example, our experiments on single-image based reconstruction methods reveal that DECA performs the best on nose regions, while GANFit performs better on cheek regions. Besides, a new and high-quality 3DMM basis, HIFI3D++, is further derived using the same procedure as we construct REALY to align and retopologize several 3D face datasets. We will release REALY, HIFI3D++, and our new evaluation pipeline at https://realy3dface.com.

Keywords: 3D Face Reconstruction, Evaluation, Benchmark, 3DMM

REALY Benchmark (Region-aware 3D face benchmark based on the LYHM dataset)

The REALY benchmark uses multi-view rendered portraits images of 100 high quality scans from the HeadSpace dataset to evaluate 3D face reconstruction accuracy. Examples of REALY benchmark: First row: globally aligned high-resolution scans with textures. Second row: retopologized meshes in HIFI3D topology with semantically consistent keypoints (red points). Third row: high quality face region masks of each scan. Fourth-Seventh row: multi-view images of each scan. Eighth row: frontal images of each scan.

The single-view reconstruction results of different methods are reported separately (100 frontal images + 400 side-view images).

The statistical information of REALY is summarized as follows.

The single-view reconstruction results of different methods are reported separately (100 frontal images + 400 multi-view images).

Reconstruction from Single Image (frontal-view)

Reconstruction from Single Image (side-view)

"*-c" indicates coarse model and "*-f" indicates fine model. bold font indicates the best one in the column. The benchmark is continuously updated to encourage researchers to participate in. We only consider methods that were defined as public when submitting to REALY, or that are publicly available in a peer-reviewed conference or journal. Methods are ranked according to the average region-wise error, you can also sort by a specified metic by clicking its corresponding table header.

We ensure results submitted to participate in the REALY benchmark are kept confidential strictly. No information about the participants or the results is released or shared with others until explicitly specified by the participants, or after appearing in a peer-reviewed conference or journal. In order to publish results to the REALY benchmark website, follow the Participation Instructions below.

Participation Instructions

To participate in the challenge, you should first get the access of REALY benchmark data according to the guideline here. Then you can use the image to reconstruct meshes and evaluate them using our evaluation pipeline.

If you want to present your method(s) on the REALY project page, please send the reconstructed meshes, barycentric coordinate and template mesh to zenghaochai@gmail.com, then we will re-evaluate and check the results. After that, we will update the project page accordingly. We recommend entitling the email as "REALY-SubmitResult-[MethodName]".

Acknowledgements and Copyrights

The benchmark is derived from the LYHM dataset. Please also cite their paper if you use this benchmark. The test portrait images are rendered using the 3D scans provided by the HeadSpace dataset. The copyright of the portraits belongs to the original owner, LYHM dataset. Please contact them to get access to the raw scan data. Note that LYHM dataset allows non-commercial research and education purposes use only.

HIFI3D++ 3DMM Basis

We also present the expressive 3DMM during constructing our REALY benchmark. The HIFI3D++ 3DMM is a full-head shape basis built from about 2,000 high quality topology consistent shapes by retopologizing HeadSpace, FaceScape, and HIFI3D. We have released HIFI3D++ at REALY!

Acknowledgements

HIFI3D++ is derived from HeadSpace, FaceScape, and HIFI3D. Please also cite these papers if you used this 3DMM basis.

Citation

If you find this work is useful for your research, please cite the following papers.

The REALY paper:

      
@inproceedings{REALY,
  title={REALY: Rethinking the Evaluation of 3D Face Reconstruction},
  author={Chai, Zenghao and Zhang, Haoxian and Ren, Jing and Kang, Di and Xu, Zhengzhuo and Zhe, Xuefei and Yuan, Chun and Bao, Linchao},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year = {2022}
}
      
    

The LYHM paper:

      
@article{LYHM,
    title={Statistical Modeling of Craniofacial Shape and Texture},
    author={Dai, Hang and Pears, Nick and Smith, William and Duncan, Christian},
    journal={International Journal of Computer Vision},
    year={2019}
}
      
    

The FaceScape paper:

      
@inproceedings{FaceScape, 
    title={FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction}, 
    author={Yang, Haotian and Zhu, Hao and Wang, Yanru and Huang, Mingkai and Shen, Qiu and Yang, Ruigang and Cao, Xun}, 
    booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, 
    year={2020} 
}
      
    

The HIFI3D paper:

      
@article{HIFI3D,
    title={High-Fidelity 3D Digital Human Head Creation from RGB-D Selfies},
    author={Bao, Linchao and Lin, Xiangkai and Chen, Yajing and Zhang, Haoxian and Wang, Sheng and Zhe, Xuefei and Kang, Di and Huang, Haozhi and Jiang, Xinwei and Wang, Jue and Yu, Dong and Zhang, Zhengyou},
    journal={ACM Transactions on Graphics},
    year={2021}
}
      
    

Contact

If you have any question, please contact Zenghao Chai or Linchao Bao.