This is a git repository for the project described here :
https://www.hackster.io/brunojje/human-pose-estimation-as-game-controller-4de02b
This project implement a server that will convert control commands to keyboard/mouse actions.
The commands are generated by an application running on a 'Kria' board that implement blazepose to generate commands from the user poses.
See the following repo for the associated client application that must be run on the Kria board.
https://github.com/BrunoJJE/bodyctrl-xlnx-kria
Performances are poor for now !
The refresh rate of the control is too slow, and suffers a large latency too.
Because of this, this can be used to test things, but you won't be able to really enjoy your game by controlling it this way for now.
You have been warned !
A PC, that will run the game that you want to control with your body.
A 'Kria' board with :
- an ethernet cable
- a webcam
- linux running on it (with keyboard and monitor)
Launch the server app on your PC :
python3 bodyctrl_pc.py
Launch the client app on the Kria board :
python3 bodyctrl_pad.py
Adjust the webcam position connected to the Kria board to be fully visible in the image displayed on the kria monitor.
When ready, press 'space' on the keyboard of the Kria board to start (or restart) the command generation (there will be a count down to let you enough time to get back in position in front of the webcam).
By default, the actions associated to the control command are not activated.
To toggle on and off the 'body control', you must stay about 10 seconds in front of the webcam in the 'vitruvian' man stand (arm and legs spread appart).
Make sure the game is launch and has focus on your PC before activating the body control !
The following actions are defined (they have been tested with the free to play game 'Destiny2' configured for keyboard/mouse control).
right : key 'D'
left : key 'A'
forward : key 'W'
backward : key 'S'
jump : key 'SPACE'
toggle sprint : key 'SHIFT'
toggle crouch : key 'CONTROL'
fire : left mouse button
The following body control command are defined (and linked to keyboard/mouse actions).
'B' : backward : feet tight
'f' : forward : feet appart
'F' : sprint forward : feet wide appart
'-' : ... : feet naturally spread
'R' : right : lean right
'L' : left : lean left
'|' : ... : straight standing, no left or right lean
'J' : jump : jump (slightly crouch then stand straigth)
'C' : crouch : slightly crouched stand
'r' : aim to the right : point arms to the right
'l' : aim to the left : point arms to the left
'S' : shot : tighten the arms (join hands)
'.' : no shot : keep your arms appart