Step by step NCS benchmarking

The entire NCSbench platform is based on Unix systems and runs on Python. Unix enables high flexibility in choosing the designs that can be deployed in the framework, and Python provides a simple and broad way to program the interfaces of our NCSbench platform.

The Robot is built by following the default instructions of the Gyro Boy robot of the Lego Mindstorms Education EV3 Core Set until step 61.

As the NCSbench platform requires a Robot (R) and a Controller (C), two devices need to be set up. We assume that the Robot runs on the Lego Mindstorms platform and the Controller on a standard PC. Additionally, we assume that an arbitrary IP network interface is available on both Controller and Robot, and used for network connection.

1a. Controller Setup

In order to collect the software requirements for the Controller, we have used the O.S Ubuntu 18.04 LTS. You can find here links on how to download and install Ubuntu 18.04 LTS on a standard PC.

Starting from a clean version of Ubuntu, the following programs need to be installed using:

C: $ sudo apt install git python3 python3-pip tcpdump ssh doxygen

Download the source code and install python-specific dependencies:

C: $ git clone $PROJECT_URL NCSbench
C: $ cd NCSbench
C: $ python3 -m venv venv
C: $ source venv/src/activate
C: $ pip3 install -r requirements-controller.txt

Finally, set up a static IP network connection on the Controller. To do this, open a terminal and run:

C: $ sudo ip addr add 192.168.10.2/24 dev en5
C: $ sudo ip link set down dev en0 && sudo ip link set up dev en5

1b. Robot Setup

In order to setup the Lego Mindstorm EV3 robot, you need to (1) download and (2) flash the ev3-jessie image on an (minimum) 8GB microSD card. We highly recommend to follow the official EV3 Getting Started guide for this operation.

Start the Robot. The first time it will not have a working network configuration. To set it up, you need to navigate the EV3 menu to > Wireless and Networks > All Network Connections and select the desired network interface from the menu. In this tutorial, we will use a static IP address configuration:

• IP address: 192.168.10.3
• Netmask: 255.255.255.0
• Gateway: 192.168.10.1

On the robot no further dependencies need to be installed.

Finally, establish a remote SSH connection to the Robot and clone the repository. In general, this can be done from any computer, and, in this tutorial, we will use the Controller to do this. As shown on the EV3 website, you can do this using the following command:

R: $ ssh [email protected] #pwd: maker
R: $ git clone $PROJECT_URL NCSbench

2. Running the NCS benchmark

Let's start by checking that the Controller and Robot are successfully connected to each other. You can use the ping command on the Controller for this:

C: $ ping 192.168.10.3

Then, start the Robot's script. To do this, navigate in the src folder and execute robot.sh. The script takes the address of the Controller as a unique argument:

R: $ cd src
R: $ sudo ./robot.sh 192.168.0.2

Now, calibrate the Robot's sensors. Lay down the Robot on its back (see picture on the left), then press the up button (N. 2) to start the calibration. When the calibration is done, the Robot will start sending sensor data to the Controller.

Now it's possible to start the benchmark experiment on the Controller. Every experiment logs data for a duration of D seconds, and stores them in the measurement folder MEAS_FOLDER. The logged data consists of real-time values of the NCS and packets of the Controller's network interface NET_IF. In our example, we store in the folder hello-world data for a duration of 60s from the Controller's network interface en5.

To start it, run the controller.sh script by specifying, additionally, the IP address of the Robot:

C: $ cd src
C: $ sudo ./controller.sh 60 hello-world 192.168.10.3  en5

Finally, lift the Robot to the vertical position to close the loop. The Controller will reply to the Robot's sensor data only when it reaches the vertical position. When this happens, the messages Control loop started. will appear on the Robot's and Controller's shell scripts.

3. Visualize the NCS benchmark results

The result of the benchmark experiment is a set of log and pcap files stored in the folder benchmark/measurement/. These logs can be processed and visualized using the jupyter notebook srip plotting.ipynb in the folder *benchmark/plots`. For executing the jupyter notebook change to the plot folder and execute the following commands:

pip3 install jupyter pandas numpy matplotlib scapy matplotlib2tikz
jupyter-notebook ./plotter.ipynb

After that the notebook is started in your browser and can be executed. The notebook script is referencing the sample measurement in ../measurement/platform-a-wireless. It takes the files contained in the measurement folder and generates plots. A sample output is available in the same folder named plotter_sample_output.html.

The sample measurements generated are the same as used in our paper.