Hao Shao, Letian Wang, RuoBing Chen, Hongsheng Li, Yu Liu
This repository contains code for the paper Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer. This work fully processes and fuses information from multi-modal multi-view sensors for achieving comprehensive scene understanding. Besides, intermediate interpretable features are generated from our framework, which provides more semantics and is exploited to better constrain actions to be within the safe sets. Our method achieved new state-of-the-art on CARLA AD Leaderboard by June 11 2022.
Install anaconda
wget https://repo.anaconda.com/archive/Anaconda3-2020.11-Linux-x86_64.sh
bash Anaconda3-2020.11-Linux-x86_64.sh
source ~/.profileClone the repo and build the environment
git clone https://github.com/opendilab/InterFuser.git
cd InterFuser
conda create -n interfuser python=3.7
conda activate interfuser
pip3 install -r requirements.txt
cd interfuser
python setup.py developDownload and setup CARLA 0.9.10.1
chmod +x setup_carla.sh
./setup_carla.sh
easy_install carla/PythonAPI/carla/dist/carla-0.9.10-py3.7-linux-x86_64.eggNote: we choose the setuptools==41 to install because this version has the feature easy_install. After installing the carla.egg you can install the lastest setuptools to avoid No module named distutils_hack.
The data is generated with leaderboard/team_code/auto_pilot.py in 8 CARLA towns using the routes and scenarios files provided at leaderboard/data on CARLA 0.9.10.1
The collected dataset is structured as follows:
- TownX_{tiny,short,long}: corresponding to different towns and routes files
- routes_X: contains data for an individual route
- rgb_{front, left, right, rear}: multi-view camera images at 400x300 resolution
- seg_{front, left, right}: corresponding segmentation images
- depth_{front, left, right}: corresponding depth images
- lidar: 3d point cloud in .npy format
- birdview: topdown segmentation images required for training LBC
- 2d_bbs_{front, left, right, rear}: 2d bounding boxes for different agents in the corresponding camera view
- 3d_bbs: 3d bounding boxes for different agents
- affordances: different types of affordances
- measurements: contains ego-agent's position, velocity and other metadata
- other_actors: contains the positions, velocities and other metadatas of surrounding vehicles and the traffic lights
In addition to the dataset, we have also provided all the scripts used for generating data and these can be modified as required for different CARLA versions. The dataset is collected by a rule-based expert agent in differnet weathers and towns.
# start 14 carla servers: ip [localhost], port [20000 - 20026]
cd carla
CUDA_VISIBLE_DEVICES=0 ./CarlaUE4.sh --world-port=20000 -opengl &
CUDA_VISIBLE_DEVICES=1 ./CarlaUE4.sh --world-port=20002 -opengl &
...
CUDA_VISIBLE_DEVICES=7 ./CarlaUE4.sh --world-port=20026 -opengl &Instructions for setting up docker are available here. Pull the docker image of CARLA 0.9.10.1 docker pull carlasim/carla:0.9.10.1.
Docker 18:
docker run -it --rm -p 2000-2002:2000-2002 --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=0 carlasim/carla:0.9.10.1 ./CarlaUE4.sh --world-port=2000 -opengl
Docker 19:
docker run -it --rm --net=host --gpus '"device=0"' carlasim/carla:0.9.10.1 ./CarlaUE4.sh --world-port=2000 -openglIf the docker container doesn't start properly then add another environment variable -e SDL_AUDIODRIVER=dsp.
Generate scripts for collecting data in batches.
cd dataset
python init_dir.py
cd ..
cd data_collection
python generate_yamls.py # You can modify fps, waypoints distribution strength ...
# If you don't need all weather, you can modify the following script
python generate_bashs.py
python generate_batch_collect.py
cd ..Run batch-run scripts of the town and route type that you need to collect.
bash data_collection/batch_run/run_route_routes_town01_long.sh
bash data_collection/batch_run/run_route_routes_town01_short.sh
...
bash data_collection/batch_run/run_route_routes_town07_tiny.shNote1: If you don't need all 14 kinds of weather in your dataset, you can modify the above code and scripts.
Note2: We also provide 7 kinds of night weather conditions in leaderboard/team_code/auto_pilot.py, you can modify the above code and scripts to collect the night dataset.
With a single CARLA server, rollout the autopilot to start data generation.
carla/CarlaUE4.sh --world-port=2000 -opengl
./leaderboard/scripts/run_evaluation.shThe expert agent used for data generation is defined in leaderboard/team_code/auto_pilot.py. Different variables which need to be set are specified in leaderboard/scripts/run_evaluation.sh.
We provide some useful python scripts in tools/data, for example:
batch_merge_data.py: merge several fragmented data items into one item, and theninterfuser/timm/data/carla_dataset.pycan be modified to reduce IO time.batch_stat_data.py: get statistical results of collected databatch_preload.py: compute the waypoints of all frames in one route and save to one filebatch_stat_blocked_data.pyandbatch_recollect_blocked_data.py: find and delete the frames that the ego-vehicle is blocked for a long time
cd interfuser
bash scripts/train.shGPU_NUM=8
DATASET_ROOT='dataset/'
./distributed_train.sh $GPU_NUM $DATASET_ROOT --dataset carla --train-towns 1 2 3 4 6 7 10 --val-towns 5 \
--train-weathers 0 1 2 3 4 5 6 7 8 9 --val-weathers 10 11 12 13 \
--model interfuser_baseline --sched cosine --epochs 25 --warmup-epochs 5 --lr 0.0005 --batch-size 16 -j 16 --no-prefetcher --eval-metric l1_error \
--opt adamw --opt-eps 1e-8 --weight-decay 0.05 \
--scale 0.9 1.1 --saver-decreasing --clip-grad 10 --freeze-num -1 \
--with-backbone-lr --backbone-lr 0.0002 \
--multi-view --with-lidar --multi-view-input-size 3 128 128 \
--experiment interfuser_baseline \
--pretrained
The DATASET_ROOT needs a file dataset_index.txt to list the traing/evaluation data. The format should be like the follow:
route_path_dir data_frames_in_this_dir
route_path_dir needs to be a relative path to
DATASET_ROOT, the training code will concat theDATASET_ROOTand route_path_dir as the full path to load the data
for example:
weather-7/data/routes_town06_long_w7_11_28_18_28_35/ 1062
weather-2/data/routes_town01_short_w2_11_16_08_27_10/ 1785
weather-2/data/routes_town01_short_w2_11_16_09_55_05/ 918
weather-2/data/routes_town02_short_w2_11_16_22_55_25/ 134
weather-2/data/routes_town01_short_w2_11_16_11_44_08/ 569 ...
1062 is the number of the frames in weather-7/data/routes_town06_long_w7_11_28_18_28_35/rgb or weather-7/data/routes_town06_long_w7_11_28_18_28_35/rgb_front etc.
Spin up a CARLA server (described above) and run the required agent. The adequate routes and scenarios files are provided in leaderboard/data and the required variables need to be set in leaderboard/scripts/run_evaluation.sh.
Update leaderboard/scripts/run_evaluation.sh to include the following code for evaluating the model on Town05 Long Benchmark.
export CARLA_ROOT=/path/to/carla/root
export ROUTES=leaderboard/data/evaluation_routes/routes_town05_long.xml
export TEAM_AGENT=leaderboard/team_code/interfuser_agent.py
export TEAM_CONFIG=leaderboard/team_code/interfuser_config.py
export CHECKPOINT_ENDPOINT=results/interfuser_result.json
export SCENARIOS=leaderboard/data/scenarios/town05_all_scenarios.jsonCUDA_VISIBLE_DEVICES=0 ./leaderboard/scripts/run_evaluation.sh
Use ROUTES=leaderboard/data/42routes/42routes.xml and SCENARIOS=leaderboard/data/42routes/42scenarios.json to run CARLA 42 routes Benchmark.
We also provide examplery model weights for direct evaluation which can be downloaded at here. The model needs to be moved to leaderboard/team_code/.
Note: The model is trained on the part of the full dataset with several towns and weathers.
This implementation is based on code from several repositories.
If you find our repo, dataset or paper useful, please cite us as
@article{shao2022interfuser,
title={Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer},
author={Hao Shao and Letian Wang and RuoBing Chen and Hongsheng Li and Yu Liu},
journal={arXiv preprint arXiv:2207.14024},
year={2022},
}All code within this repository is under Apache License 2.0.