A combined ROS/ROS2 driver for event based cameras using Prophesee's Metavision SDK. This driver is not written or supported by Prophesee.
If you are looking for more speed and features than the official Prophesee ROS driver you have found the right repository. This driver can cope with the large amount of data produced by Prophesee's Gen3 and Gen4 sensors because it does little more than getting the RAW (currently EVT3 format) events from the camera and publishing them in ROS event_array_msg format.
The events can be decoded and displayed using the following ROS/ROS2 packages:
- event_array_codecs has C++ routines to decode event_array_msgs.
- event_array_py module for fast event decoding in python.
- event_array_viewer a node / nodelet that renders and publishes ROS image messages.
- event_array_tools
a set of tools to echo, monitor performance and convert
event_array_msgs
to legacy formats and into "RAW" format.
Tested on the following platforms:
- ROS Noetic
- ROS2 Galactic / Humble
- Ubuntu 20.04, 22.04 LTS
- Metavision SDK (OpenEB) 2.2.2 - 4.0.1
Tested on the following hardware:
Prerequisites:
- install Metavision SDK or OpenEB
- install
vcs
(ubuntu package python3-vcstool).
Make sure you have your ROS1 or ROS2 environment sourced such that ROS_VERSION is set. For example for ROS1 noetic:
source /opt/ros/noetic/setup.bash
Create a workspace (metavision_driver_ws
), clone this repo, and use vcs
to pull in the remaining dependencies:
pkg=metavision_driver
mkdir -p ~/${pkg}_ws/src
cd ~/${pkg}_ws
git clone https://github.com/ros-event-camera/metavision_driver.git src/${pkg}
cd src
vcs import < ${pkg}/${pkg}.repos
cd ..
Optional (ROS1): to use the efficient recording nodelet clone the nodelet_rosbag repository into our src tree:
git clone -b open_bag_later [email protected]:berndpfrommer/nodelet_rosbag.git src/nodelet_rosbag
Now configure and build:
ROS1:
catkin config -DCMAKE_BUILD_TYPE=RelWithDebInfo # (optionally add -DCMAKE_EXPORT_COMPILE_COMMANDS=1)
catkin build
. devel/setup.bash
ROS2:
colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
. install/setup.bash
This driver differs from the Prophesee ROS driver in the following ways:
- publishes event_array_msg that store more densely and are faster to access than the older message formats.
- less CPU consumption by avoiding unnecessary memory copies.
- implemented as nodelet such that it can be run in the same address space as e.g. a rosbag record nodelet without worrying about message loss in transmission.
- prints out message rate statistics so you know when the sensor saturates bandwidth.
- supports these additional features:
- dynamic reconfiguration for bias parameters
- ROI specification
- camera synchronization (stereo)
- external trigger events
- event rate control
Parameters:
bias_file
: path to file with camera biases. See example in thebiases
directory.from_file
: path to Metavision raw file. Instead of opening camera, driver plays back data from this file.serial
: specifies serial number of camera to open (useful for stereo). To learn serial number format first start driver without specifying serial number and look at the log files.event_message_time_threshold
: (in seconds) minimum time span of events to be aggregated in one ROS event message before message is sent. Defaults to 1ms.event_message_size_threshold
: (in bytes) minimum size of events (in bytes) to be aggregated in one ROS event message before message is sent. Defaults to 1MB.statistics_print_interval
: time in seconds between statistics printouts.send_queue_size
: outgoing ROS message send queue size (defaults to 1000 messages).use_multithreading
: decouples the SDK callback from the processing to ensure the SDK does not drop messages (defaults to false). The SDK already queues up messages but there is no documentation on the queue size and no way to determine if messages are dropped. Use multithreading to minimize the risk of dropping messages. However, be aware that this incurs an extra memory copy and threading overhead, raising the maximum CPU load by about 50% of a CPU.frame_id
: the frame id to use in the ROS message headerroi
: sets hardware region of interest (ROI). You can set multiple ROI rectangles with this parameter by concatenation:[top_left_x_1, top_left_y_1, width_1, height_1, top_left_x_2, top_left_y_2, width_2...]
. The length of theroi
parameter vector must therefore be a multiple of 4. Beware that when using multiple ROIs, per Metavision SDK documentation: "Any line or column enabled by a single ROI is also enabled for all the other".erc_mode
: event rate control mode (Gen4 sensor):na
,disabled
,enabled
. Default:na
.erc_rate
: event rate control rate (Gen4 sensor) events/sec. Default: 100000000.sync_mode
: Used to synchronize the time stamps across multiple cameras (tested for only 2). The cameras must be connected via a sync cable, and two separate ROS driver nodes are started, see example launch files. Theprimary
node'sready
topic must be remapped so it receives thesecondary
node'sready
messages. Allowed values:standalone
(default): freerunning camera, no sync.primary
: camera that drives the sync clock. Will not start publishing data until it receives aready
message from the secondary.secondary
: camera receiving the sync clock. Will sendready
messages until it receives a sync signal from the primary.
trigger_in_mode
: Controls the mode of the trigger input hardware. Allowed values:disabled
(default): Does not enable this functionality within the hardwareexternal
: Enables the external hardware pin to route to the trigger input hardware. This will be the pin on the camera's connector.loopback
: Connects the trigger out pin to the trigger input hardware. Only available on Gen3 sensors (SilkyEVCam VGA).
trigger_out_mode
: Controls the mode of the trigger output hardware. NOTE: 4-th gen sensors no longer support trigger out! Allowed values:disabled
(default): Does not enable this functionality within the hardwareenabled
: Enables the external hardware pin to route to the trigger in hardware.
trigger_out_period
: Controls the period in microseconds of the trigger out pulse.trigger_out_duty_cycle
: Controls the duty cycle of the trigger out pulse. This is the period ratio.
Services:
save_biases
: write out current bias settings to bias file. For this to work thebias_file
parameter must be set to a non-empty value.
Dynamic reconfiguration parameters (see MetaVision documentation here):
bias_diff
(read only)bias_diff_off
bias_diff_on
bias_fo
bias_hpf
bias_pr
bias_refr
roslaunch metavision_driver driver_node.launch # (run as node)
roslaunch metavision_driver driver_nodelet.launch # (run as nodelet)
The driver should print out message rate statistics like this:
[ INFO] [1663845360.494651335]: /event_camera: bw in: 106.08359 MB/s, msgs/s in: 51930, out: 0
Prints out the incoming (from the SDK) bandwidth and incoming and
published message rate. In multithreaded mode there will also be shown
the maximum queue size observed during
statistics_print_interval
.
To use the combined driver/recording facility:
roslaunch metavision_driver recording_driver.launch bag:=`pwd`/test.bag
Then start/stop recording like this:
rosrun metavision_driver start_recording.py
rosrun metavision_driver stop_recording.py
To visualize the events, run a viewer
node from the
event_array_viewer package:
roslaunch event_array_viewer viewer.launch
The viewer node publishes an image that can be visualized with e.g. rqt_image_view
For efficient recording of the events you need to run the driver and the recorder in the same address space as ROS2 composable nodes. For this you will need to install the composable recorder into your workspace as well (see below).
ros2 launch metavision_driver driver_node.launch.py # (run as node)
ros2 launch metavision_driver driver_composition.launch.py # (run as composable node)
The printout should be similar to the one for ROS1.
To use the combined driver/recorder and start the recording:
ros2 launch metavision_driver recording_driver.launch.py
ros2 run rosbag2_composable_recorder start_recording.py
To stop the recording you have to kill (Ctrl-C) the recording driver.
To visualize the events, run a viewer
node from the
event_array_viewer package:
ros2 launch event_array_viewer viewer.launch.py
The viewer node publishes an image that can be visualized with e.g. rqt_image_view
Here are some approximate performance numbers on a 16 thread (8-core) AMD Ryzen 7480h laptop with max clock speed of 2.9GHz. All numbers were obtained by producing maximum event rates about (48Mevs) with a SilkyEVCam:
All CPU loads below are with sensor saturating at close to 50Mevs.
settings | single threaded | multi threaded | note |
---|---|---|---|
driver no subscriber | 22% | 59% | no pub, extra copy for multithreaded |
driver with subscriber | 37% | 61% | does interprocess communication |
driver + rosbag record node | 70% | 96% | combined driver + record cpu load |
driver + rosbag record nodelet | 52% | 74% | single process no ipc but disk/io |
All CPU loads below are with sensor saturating at close to 50Mevs. Middleware used was cyclonedds.
settings | single threaded | multi threaded | note |
---|---|---|---|
driver no subscriber | 22% | 59% | no pub, extra copy for multithreaded |
driver with subscriber | 35% | 44% | does interprocess communication |
driver + rosbag record node | 80% | 90% | combined driver + record cpu load |
driver + rosbag record composable | 58% | 80% | single process no ipc but disk/io |
The SDK provides hardware event time stamps directly from the camera. For efficiency reasons the packets are not decoded and so the sensor time stamps are not available to the driver. Therefore the ROS driver simply puts the host wall clock arrival time of the first SDK packet into the ROS packet's header stamp field.
External triggers on prophesee cameras allows for a signal to be injected into the event stream. This is useful for synchronizing external devices. The event stream contained in the packages will now contain trigger events that can be recovered with the decoder.
Prophesee provides documentation on the trigger functions at a high level here.
Trigger out
functionality is exposed through trigger_out_mode
,
trigger_out_period
, and trigger_out_duty_cycle
. These
variables follow the same meaning as laid out in the internal
documentation.
trigger_out_mode
can beenabled
ordisabled
trigger_out_period
can be from 2us to 1h (units are us)trigger_out_duty_cycle
is the pulse width ratio (trigger_out_period * trigger_out_duty_cycle
must be at least 1us)
Trigger in
functionality is abstracted away from pins to just loopback
or
external
as the pin mappings are constant for a given camera
configuration.
trigger_in_mode
allows the user to specify for each cameraloopback
orexternal
and lookup which pins are associated with that camera.
WARNING Running synchronization and triggers at the same time is possible, but requires understanding of your camera's underlying hardware (as most share trigger out and sync out pins).
The hardware configuration file is config/trigger_pins.yaml
. The
mappings for hal_plugin_gen*
come from
Prophesee
documentation.
The mapping for evc3a_plugin_gen31
has been validated on the
SilkyEvCam (March 2022). The mapping goes from the HAL Software Info
to pin numbers.
If your camera is not yet supported, the software info is printed out on driver startup. Look for a line that contains:
Plugin Software Name:
This will be the key to place under prophesee_pin_config
which can
then be populated based on your camera's documentation.
WARNING If this file is not loaded (or your camera is not yet supported), the default pin loaded will be 0. This may work in some cases, but not all.
Documentation on the SilkyEvCam pinout can be found here on page 6. This system uses 3.3V logic for both trigger in as well as trigger out.
While the loopback configuration is internal to the chip, the trigger out line will still pulse externally. This is useful if using an event camera to trigger an external system as you will maintain the timing relative to the internal clock (after association between the trigger event and the external system).
External triggering works on SilkyEVCam HD, i.e. you can trigger the SilkyEVCam HD externally the same way as the SilkyEVCam VGA.
This software is issued under the Apache License Version 2.0.