Robot Learning of Shifting Objects for Grasping in Cluttered Environments

We've released our easy-to-use Python package Griffig!
You can find more information in its repository and on the website griffig.xyz

---

This repository contains additional information for the paper Robot Learning of Shifting Objects for Grasping in Cluttered Environments accepted for IROS 2019 in Macau. This code lets a robot learn how to grasp objects out of a bin by itself. As traditional approached oftentimes need the 3d model of the object, the robot in this project learns grasping in a self-supervised manner by try and error. Our focus relies on the data-efficiency of the learning process: Currently, it needs around 20000 grasp and around 3000 shift attempts to reliably empty a bin with a grasp rate of over 95%. Shifting is essential for bin picking, as it allows the robot to empty a bin completely.

Click the image for a video!

Our overall setup of a Franka Panda robotic arm including the standard force-feedback gripper, an Ensenso stereo camera, custom 3D-printed gripper jaws with anti-slip tape, and two industrial bins with objects. The robot learns first grasping and then shifting objects to explicitly increase grasp success. The first sections give a short introduction into the source code. Later, we present more information about the paper, i.a. a more detailed evaluation.

Structure

The overall structure is as follows:

• Database Server There is a database server for collecting and uploading data and images to another computer. The server has a web interface for showing all actions for all databases in a given server and to show the latest action live.
• Include / Src The main C++ project for controlling the robot and infer the next best action. It uses the TensorFlow C++ API and MoveIt! for robot control. The camera driver is also included, either via direct access or and optional ensenso node. The latter is helpful because the camera needs a long time to connect and crashes sometimes afterwards.
• Scripts It is recommended to export to PYTHONPATH in .bashrc: export PYTHONPATH=$PYTHONPATH:$HOME/Documents/bin_picking/scripts
• Jupyter For neural network definition and training.

This project is a ROS package with launch files and a package.xml. The ROS node /move_group is set to respawn=true. This enables to call rosnode kill /move_group to restart it.

Running the Demo

After installing all dependencies (see next section), run both roslaunch bin_picking moveit.launch and roslaunch bin_picking bin_picking.launch. For recording, check the database server and the corresponding web interface.

Models

• TensorFlow models (via the tf.saver API) for the seperated grasping and pushing NN are in the models directory.
• CAD models of the 3d-printed robotic gripper and the camera mount are in the cad-models directory

Installation

For the robotic hardware, make sure to load launch/gripper-config.json as the Franka end-effector configuration. Currently, following dependencies need to be installed:

• ROS Kinetic
• libfranka & franka_ros
• TensorFlow (with C++)
• EnsensoSDK
• Eigen unsupported (Euler angles, 3.4.0 dev)
• yaml-cpp
• Cereal
• curl & cpr
• Catch2