README.md

EgoPose

This repo contains the official implementation of our paper:

Ego-Pose Estimation and Forecasting as Real-Time PD Control
Ye Yuan, Kris Kitani
ICCV 2019
[website] [paper] [video]

Installation

Dataset

• Download the dataset from Google Drive in the form of a single zip or split zips (or BaiduYun link, password: ynui) and place the unzipped dataset folder inside the repo as "EgoPose/datasets". Please see the README.txt inside the folder for details about the dataset.

Environment

• Supported OS: MacOS, Linux
• Packages:
  • Python >= 3.6
  • PyTorch >= 0.4 (https://pytorch.org/)
  • gym (https://github.com/openai/gym)
  • mujoco-py (https://github.com/openai/mujoco-py) (MuJoCo is now free thanks to DeepMind! 🎉🎉🎉)
  • OpenCV: conda install -c menpo opencv
  • Tensorflow, OpenGL, yaml: conda install tensorflow pyopengl pyyaml
• Additional setup:
  • For linux, the following environment variable needs to be set to greatly improve multi-threaded sampling performance:
    export OMP_NUM_THREADS=1
• Note: All scripts should be run from the root of this repo.

Pretrained Models

• Download our pretrained models from this link (or BaiduYun link, password: kieq) and place the unzipped results folder inside the repo as "EgoPose/results".

Quick Demo

Ego-Pose Estimation

• To visualize the results for MoCap data:
  python ego_pose/eval_pose.py --egomimic-cfg subject_03 --statereg-cfg subject_03 --mode vis
  Here we use the config file for subject_03. Note that in the visualization, the red humanoid represents the GT.

• To visualize the results for in-the-wild data:
  python ego_pose/eval_pose_wild.py --egomimic-cfg cross_01 --statereg-cfg cross_01 --data wild_01 --mode vis
  Here we use the config file for cross-subject model (cross_01) and test it on in-the-wild data (wild_01).

• Keyboard shortcuts for the visualizer: keymap.md

Ego-Pose Forecasting

• To visualize the results for MoCap data:
  python ego_pose/eval_forecast.py --egoforecast-cfg subject_03 --mode vis

• To visualize the results for in-the-wild data:
  python ego_pose/eval_forecast_wild.py --egoforecast-cfg cross_01 --data wild_01 --mode vis

Training and Testing

• If you are interested in training and testing with our code, please see train_and_test.md.

Citation

If you find our work useful in your research, please cite our paper Ego-Pose Estimation and Forecasting as Real-Time PD Control:

@inproceedings{yuan2019ego,
    title={Ego-Pose Estimation and Forecasting as Real-Time PD Control},
    author={Yuan, Ye and Kitani, Kris},
    booktitle={Proceedings of the IEEE International Conference on Computer Vision (ICCV)},
    year={2019},
    pages={10082--10092}
}

License

The software in this repo is freely available for free non-commercial use. Please see the license for further details.