- GARDE Guillaume (Promotion ENSTA Bretagne 2024 - Spécialité Robotique Autonome).
- MASSA Théo (Promotion ENSTA Bretagne 2024 - Spécialité Robotique Autonome).
This project was made during an internship at the University of Amsterdam under the supervision of Arnoud Visser PhD.
This repository provides tools to set up a ROS 2 interface for controlling the UMI-RTX robotic arm. The mission of this project is to make the arm detect a target (we chose a banana plush) via computer vision and then move to grab it and lift it. The different nodes of the ROS architecture correspond to:
- inverse kinematics
- arm control
- computer vision
- simulation
- custom Graphic User Interface (GUI)
A vast majority of this project's code (the one we wrote) is in C++.
A documentation of the code can be found in /ROS_ws/doc/. To open the doc in html, just launch index.html.
You can find the original drivers in /umi-rtx or on this repository
Here is the internship report related to this project.
This project is built and tested with Ubuntu 20.04 and ROS2 Foxy. In case these settings are not supported, see this part.
Used material in this project:
-
A UMI-RTX robotic arm
-
A Stereolabs ZED Mini stereo camera and its cable
-
A standard banana plush
-
An Intel Core i9 processor and a Nvidia GPU (Geforce RTX)
Note: The ZED Mini cable needs to be plugged into the device with its incurved arrows on the same side as the lenses.
The computer vision node uses OpenCV and Stereolabs' Software Development Kit (SDK). To use the SDK, a powerful Nvidia Graphics Processing Unit (GPU) is required. See here for a compatibility check and installation.
Note: Two versions of the computer vision node are provided here but only one will be launched. This version uses the SDK and OpenCV. The other version is the first we wrote and only uses OpenCV. However, it is not performant enough and gives bad results.
If you plan not to use Docker, you need to download and install the SDK as well as Nvidia drivers. See here to install the SDK on Linux. If you followed all the steps, the Nvidia drivers should be installed.
In order to use the interface, you will need to build the ROS 2 package (don't forget to source ROS 2 beforehand).
## install the required dependencies
./install_dependencies.sh
cd ROS_ws
## Build the package
colcon build
If you are on bash:
source install/setup.bash
If you are on zsh:
source install/setup.zsh
To test the computer vision node separatly, follow the commands below:
ros2 run ros_interface_umi_rtx nodeCameraAPI
Once the node is running, you can interact with its output thanks to diverses tools like RQt or RViz.
To use the interface, all you have to do is :
cd LAB42_RTX_control
## Launch the arm
./start_arm
The custom GUI will launch in the same time as the arm
If you want to monitor the nodes and topics or interact with them, all you have to do is logging as root:
sudo -i
And then sourcing ROS:
source /opt/ros/foxy/setup.bash
In order to start only the simulation, just run:
ros2 launch ros_interface_umi_rtx simu.launch.py
As told earlier, this project works under Ubuntu 20.04 and ROS2 Foxy. However, the usage of the interface is not limited to this configuration thanks to a custom Docker image that allows to use our interface with a different configuration.
The only requirements are to have Docker installed on your computer (see here to install Docker), to have a NVIDIA GPU with the necessary drivers for which the installation process is described earlier, and having installed the nvidia-docker-toolkit.
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update && sudo apt-get install -y nvidia-docker2 nvidia-container-toolkit
sudo systemctl daemon-reload
sudo systemctl restart docker
Once this is done, you can build the docker image with the command:
# Place yourself in LAB42_RTX_CONTROL
docker build -t "name" .
Be careful to replace "name" with the name you want, and everything is ready !
To run our image into a container, run :
# Give the permission to use the screen
xhost +
# Launch the container (replace "name" with the name you chose)
docker run --gpus all -it --privileged -e DISPLAY=$DISPLAY -v \\
/tmp/.X11-unix:/tmp/.X11-unix --rm "name":latest
Once the container is running, the process is similar as the one descibed here
cd ROS_ws/
colcon build
source install/setup.bash
cd ..
If you want to use the arm, do
./start_arm.sh
If you want to launch only the simulation, do
ros2 launch ros_interace_umi_rtx simu.launch.py
Here is a link to a Youtube playlist where you can find some trials and demonstration of the interface.
An other way to manage the computer vision part is to have a look at the following files:
ROS_ws/src/ros_interface_umi_rtx/include/ros_interface_umi_rtx/node_camera.hpp
ROS_ws/src/ros_interface_umi_rtx/src/node_camera.cpp
This code does not require the SDK to work but needs to be improved and corrected.
In the CMakeLists.txt file, comment these two lines
find_package(ZED 3 REQUIRED)
find_package(CUDA ${ZED_CUDA_VERSION} REQUIRED)
In include_directories(), also comment
${CUDA_INCLUDE_DIRS}
${ZED_INCLUDE_DIRS}
Comment this whole section
##################################### Node camera API
add_executable(nodeCameraAPI
src/node_camera_API.cpp
)
ament_target_dependencies(nodeCameraAPI
rclcpp
sensor_msgs
OpenCV
cv_bridge
geometry_msgs
ament_index_cpp
std_msgs
ZED
CUDA
)
target_link_libraries(nodeCameraAPI "${cpp_typesupport_target}" ${OpenCV_LIBS} ${ZED_LIBRARY_DIR} ${CUDA_LIBRARY_DIRS})
target_include_directories(nodeCameraAPI PUBLIC
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
$<INSTALL_INTERFACE:include>)
Replace the line
install(TARGETS nodeCamera nodeArm nodeInverseKinematics nodeSimu GUI nodeCameraAPI
DESTINATION lib/${PROJECT_NAME})
By
install(TARGETS nodeCamera nodeArm nodeInverseKinematics nodeSimu GUI
DESTINATION lib/${PROJECT_NAME})
In package.xml, comment
<depend>sl</depend>
In the launch files
ROS_ws/src/ros_interface_umi_rtx/launch/arm.launch.py
ROS_ws/src/ros_interface_umi_rtx/launch/simu.launch.py
Delete
nodeCameraAPI = Node(
package = 'ros_interface_umi_rtx',
namespace='',
executable='nodeCameraAPI',
name='camera_api',
)
Then replace nodeCameraAPI by nodeCamera in return LaunchDescription( ... ). You are now all set and can follow the building section.