Graph Convolutional Networks for Assessment of Physical Rehabilitation Exercises

This code is the official implementation of the following works (train + eval):

• S. Deb, M. F. Islam, S. Rahman and S. Rahman, "Graph Convolutional Networks for Assessment of Physical Rehabilitation Exercises," in IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 30, pp. 410-419, 2022, doi: 10.1109/TNSRE.2022.3150392.

Figure 1: Overview of existing vs. the proposed method. (a) The existing deep learning method applies CNN to the grid structure of stacked skeleton (body-joints) data. It performs consistently only with fixed-length input and ignores spatio-temporal topological structure from interaction among neighborhood joints. (b) Our proposed method employs STGCN to address the issues mentioned above. We offer extensions to STGCN using LSTM to extract rich spatio-temporal features and attend to different body-joints (as illustrated in colored joints) based on their role in the given exercise. It enables our method to guide users for better assessment scores.

Figure 2: GCN based end-to-end models using (a-b) vanilla STGCN and (c-d) extended STGCN for rehabilitation exercise assessment. 'TC', \oplus and \odot denote temporal convolution, concatenation and element-wise multiplication, respectively. (b) and (d) illustrate the detailed components of the green STGCN block of (a) and (c), respectively.

Data Preparation

We experimented on two skeleton based rehabilitation datasts: KIMORE and UI-PRMD. Before training and testing, for the convenience of fast data loading, the datasets should be converted to the proper format. Please download the pre-processed UI-PRMD dataset from here GoogleDrive and preprocess KIMORE dataset from here GoogleDrive. Extract files with command below.

cd st-gcn
unzip <path to Dataset.zip>

Requirements

• Python3 (>3.5)
• Install Tensorflow 2.0 from https://www.tensorflow.org/install
• To install other libraries simply run pip install -r requirements.txt

Files

• train.py : to perform training on Physical rehabilitation exercise
• data_preprocessing.py : preproces the data collected from dataset. It is mandatory to do some preprocessing before feeding it network.
• graph.py : It will generate skeleton graph from given data
• stgcn_lstm.py : build propose ST-GCN method
• demo.py : perform a demo inference for given sample.

Running instructions

You can use the following commands to run the demo.

python demo.py [--skeleton data ${PATH_TO_DATA}] [--label ${PATH_TO_Label}]

# Alternative way
python demo.py

The output is the predicted label for the demo exercise.

To train the model you have to first download the dataset from above link. The data and labels of an exercise have to be inside a folder. Then run the train.py file to train the model. You can change the optimizer, learning rate and other parameters by editing train.py. The total number of training epoch is 2000; the learning rate is initialized as 0.0001. You can train the model following command.

python train.py --ex Kimore_ex5 --epoch 2000 --batch_size 10

Notes on experiment

Citation

If you find our research helpful, please consider cite this work:

@ARTICLE{deb-2022-graph,
  author={Deb, Swakshar and Islam, Md Fokhrul and Rahman, Shafin and Rahman, Sejuti},
  journal={IEEE Transactions on Neural Systems and Rehabilitation Engineering}, 
  title={Graph Convolutional Networks for Assessment of Physical Rehabilitation Exercises}, 
  year={2022},
  volume={30},
  number={},
  pages={410-419},
  doi={10.1109/TNSRE.2022.3150392}}

Acknowledgment

We thank the authors and contributors of original GCN implementation.

Contact

For any question, feel free to contact @

Swakshar Deb     : [email protected]
Md Fokhrul Islam : [email protected]