Our wearable motion capture system provides arm-pose estimations from a single smartwatch. This allows motion capture of the human arm anytime and anywhere. Our system comprises apps to stream sensor data from wearable devices, a python package to estimate arm poses from streamed sensor data, and an optional 3D visualization using Unity.
📱⌚ 💻 Please visit this website for videos and more material.
The associated paper to this system.
@INPROCEEDINGS{iroco2024,
author={Weigend, Fabian C and Liu, Xiao and Sonawani, Shubham and Kumar, Neelesh and Vasudevan, Venugopal and Ben Amor, Heni},
booktitle={2024 IEEE International Conference on Robotics and Automation (ICRA)},
title={iRoCo: Intuitive Robot Control From Anywhere Using a Smartwatch},
year={2024},
pages={17800-17806},
doi={10.1109/ICRA57147.2024.10610805}
}
For more detailed instructions and advanced modes please see our wiki. As shown below, this framework features three modes. Watch Only estimates arm poses from the data of a single smartwatch. +Phone Upper Arm enables our most accurate arm pose predictions. +Phone Pocket allows to move around and change body orientation. The quick start guide gets you started with the Watch Only mode.
Our wearable motion capture system is composed of modules. Each of these modules is one repository in this GitHub Organization. Namely, they are:
- sensor-stream-apps provides the apps to stream sensor readings from wearable devices to a remote machine.
- arm-pose-estimation opens sockets to receive streamed data from wearable devices and processes them into arm posture estimations.
- arm-pose-visualization visualizes arm posture estimations in real time using a 3D avatar.
The figure below summarizes the data stream from smart devices to visualization. The individual components are connected through UDP streams, but Arm Pose Estimation and Arm Pose Visualization can run on the same machine. The UDP streaming from smart devices to the Arm Pose Estimation requires a local WiFi connection.
First, install our sensor-stream-apps with Android Studio. Your smartwatch has to be in developer mode and has to be connected the same WiFi as the machine that you want to run arm pose estimations on. If you are unfamiliar with installing an app in developer mode, we recommend following the step-by-step instructions in our extensive wiki.
Then, clone arm-pose-estimation and install it with pip:
pip3 install /path/to/cloned/repository
For streaming data from the watch only, copy the watch_only.py from the example_scripts directory. Run the script with your local IP as a parameter. For example:
python3 watch_only.py 192.168.1.123
Finally, to visualize arm pose estimates, you can either download a build of the visualizer tool here, or you clone our arm-pose-visualization repository and open it in Unity. Please also see our wiki if you want to use Unity.
