This software implements the Social Pattern Extraction Convolution (SPEC) algorithm published in the following paper:
Dapeng Zhao and Jean Oh, Noticing Motion Patterns: A Temporal CNN With a Novel Convolution Operator for Human Trajectory Prediction, IEEE Robotics and Automation Letters (RA-L), December 2020.
@article{zhao2020noticing, title={Noticing Motion Patterns: Temporal CNN with a Novel Convolution Operator for Human Trajectory Prediction}, author={Zhao, Dapeng and Oh, Jean}, journal={IEEE Robotics and Automation Letters}, year={2020}, publisher={IEEE} }
Social Pattern Extraction Convolution (SPEC)[1] is an algorithm designed and implemented to help robots to predict humans’ future trajectories, so that robots can navigate in a safe and self-explanatory manner, while humans feel comfortable about robots’ motion.
System Functionality:
- Can be trained on the ETH pedestrian dataset[2] and UCY pedestrian dataset[3], which are provided together with the software in the “./data” folder
- Can be trained on other datasets, as long as given in the same format as the provided ETH/UCY datasets
- Can predict future trajectories of human pedestrians, when given observed history trajectories
- Can be used to simulate human behaviors, when given appropriate initialization, e.g. a few steps of locations, which is useful for social navigation study
- Can evaluate given trajectories, i.e. report their similarities to the trajectories used for training, which can potentially be useful to surveillance and anomaly detection study
Dapeng Zhao
PI: Jean Oh
git clone https://github.com/cmubig/SPEC.git
cd SPEC
- numpy==1.18.2+
- torch==1.4.0
These are the specifications of the hardware used during development.
Different combinations of hardware could work and will yield different speeds.
- Intel(R) Core(TM) i9-9820X CPU @ 3.30GHz
- 16GB ram x 8
- GeForce RTX 2080 Ti
These are the specifications of the software used during development. Different combinations of software might work.
- Ubuntu 18.04
- CUDA 11.0
Below are some empirical observation based non-functional expectations which users can have when trying to utilize the provided software:
- Default training procedure (Dataset: zara1, number of epochs: 100, batch size: 64) could take ~180 seconds
- Inference time of one trajectory is ~5 milliseconds
To train, run:
python train.py
To train with different setting, add a token in Arguments.py with your desired setting, and run, e.g. token as "hotel":
python train.py --token "hotel"
Functions for inference can be found in model.py.
This work was supported in part by the U.S. Air Force Office of Scientific Research under Award no. FA2386-17-1-4660 and in part by U.S. Army Ground Vehicle Systems Center.
[1] D. Zhao and J. Oh, "Noticing Motion Patterns: A Temporal CNN With a Novel Convolution Operator for Human Trajectory Prediction," in IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 628-634, April 2021, doi: 10.1109/LRA.2020.3047771.
https://ieeexplore.ieee.org/abstract/document/9309403/
[2] S. Pellegrini, A. Ess, K. Schindler and L. van Gool, "You'll never walk alone: Modeling social behavior for multi-target tracking," 2009 IEEE 12th International Conference on Computer Vision, Kyoto, Japan, 2009, pp. 261-268, doi: 10.1109/ICCV.2009.5459260.
https://ieeexplore.ieee.org/abstract/document/5459260
[3] A. Lerner, Y. Chrysanthou, and D. Lischinski, “Crowds by example,” in Computer graphics forum, vol. 26, no. 3. Wiley Online Library, 2007, pp. 655–664.
https://onlinelibrary.wiley.com/doi/full/10.1111/j.1467-8659.2007.01089.x