SPARK: Self-supervised Personalized Real-time Monocular Face Capture

Kelian Baert . Shrisha Bharadwaj · Fabien Castan · Benoit Maujean . Marc Christie · Victoria Fernandez Abrevaya . Adnane Boukhayma

Technicolor | Max Planck Institute | INRIA Rennes
SIGGRAPH Asia 2024 Conference Papers

Citation

If you find our code or paper useful, please cite as:

@inproceedings{baert2024spark,
  title = {{SPARK}: Self-supervised Personalized Real-time Monocular Face Capture},
  author = {Baert, Kelian and Bharadwaj, Shrisha and Castan, Fabien and Maujean, Benoit and Christie, Marc and Abrevaya, Victoria and Boukhayma, Adnane},
  booktitle = {SIGGRAPH Asia 2024 Conference Proceedings},
  doi = {10.1145/3680528.3687704},
  isbn = {979-8-4007-1131-2/24/12},
  month = dec,
  year = {2024},
  url = {https://kelianb.github.io/SPARK/},
}

Installation

Details

• Create the environment using setup.sh.
• Run TrackerAdaptation/setup_submodules.sh. This may take a few minutes.
• Link FLAME from MultiFLARE to EMOCA: ln TrackerAdaptation/submodules/EMOCA/assets/FLAME/geometry/generic_model.pkl MultiFLARE/assets/flame/flame2020.pkl
  • This is equivalent to downloading FLAME (2020 version), unzipping it and copying generic_model.pkl at ./MultiFLARE/assets/flame/flame2020.pkl.
• Get Basel Face Model texture space adapted to FLAME. Unfortunately, we are not allowed to distribute the texture space since the license does not permit it. Therefore, please go to the BFM page, sign up and download BFM. Then use the tool from this repo to convert the texture space to FLAME. Put the resulting texture model file at TrackerAdaptation/submodules/EMOCA/assets/FLAME/texture/FLAME_albedo_from_BFM.npz.

SPARK has been tested with NVIDIA RTX A5000 (24GB) or RTX A4000 (16GB) GPUs. It is possible to train on GPUs with less memory by reducing the batch size.

Dataset

Please refer to the MonoFaceCompute repository to preprocess your own data.

Usage

SPARK is a two-stage approach. First, run MultiFLARE to reconstruct a 3D Face Avatar from multiple videos. Then, use TrackerAdaptation to adapt an existing 3D face tracker to your avatar for real-time tracking through transfer learning.

Details

1. MultiFLARE

cd MultiFLARE
python train.py --config configs/example.txt)

We advise starting from the provided example config and modifying input_dir, train_dir and output_dir. For a list of all parameters, please refer to arguments.py or the output of python train.py --help. Parameters can be passed either in the config file or as command line arguments.

2. TrackerAdaptation

cd TrackerAdaptation
# DECA encoder + MultiFLARE decoder
python train.py --config configs/example_deca.txt
# EMOCA encoder + MultiFLARE decoder
python train.py --config configs/example_emoca.txt
# SMIRK encoder + MultiFLARE decoder (recommended!)
python train.py --config configs/example_smirk.txt
# EMOCA encoder + EMOCA decoder (baseline)
python train.py --config configs/example_emoca_baseline.txt

# Quantitative eval
python evaluate.py --config configs/example_smirk.txt --tracker_resume 3000 --frame_interval 5 --num_frames 64

# Visualization videos
python make_comparison_video.py --config configs/example_smirk.txt --tracker_resume 3000 --test_dirs 5 6 --n_frames 1000 --smooth_crops --framerate 24
python make_overlay_video.py --config configs/example_smirk.txt --tracker_resume 3000 --test_dirs 2 --out test_beard --texture /path/to/texture.png --n_frames 1000 --smooth_crops --framerate 24

License Information

The code in this repository is subject to multiple licenses.

1. Original Code (Technicolor Group & INRIA Rennes)

   • All code in this repository, except where otherwise specified, is licensed under the CC BY-NC-SA License.

2. Third-Party Code (Max Planck Institute for Intelligent Systems)

   • Location: ./MultiFLARE/flame, ./MultiFLARE/flare, ./TrackerAdaptation/submodules
   • These directories contain code by Max Planck Institute, with some modifications. Please carefully read the MPI License and note that this is only available for non-commercial scientific research purposes.