Learning to Predict Navigational Patterns from Partial Observations

Code accompanying the paper "Learning to Predict Navigational Patterns from Partial Observations" (RA-L 2023).

The paper presents a self-supervised method to learn navigational patterns in structured environments from partial observations of other agents. The navigational patterns are represented as a directional soft lane probability (DSLP) field. We also present a method for inferring the most likely discrete path or lane graph based on the predicted DSLP field.

Paper link: Predictive World Models from Real-World Partial Observations

Video presentation link: TODO

Data directory link: Google Drive directory

Installation

The code is tested with Python 3.9 on Ubuntu 22.04.

Download all submodules

git submodule update --init --recursive

The submodules are used for the following tasks

1. predictive-world-models: Predictive world model repository
2. vdvae: Code for implementing the predictive world model. Fork of the original VDVAE repository modified to a dual encoder posterior matching HVAE model.

Install dependencies

Follow README instructions in predictive-world-models/

Downgrade Pytorch Lightning --> 1.9.0 (for CLI implementation to work)

pip uninstall pytorch-lightning
pip install pytorch-lightning==1.9.0

Evaluation data

Download and extract the following compressed directories into the local data/ directory.

Data directory

dslp/
└───data/
    |   bev_nuscenes_256px_v01_boston_seaport_unaug_gt_full_eval_preproc.tar.gz
    |   bev_nuscenes_256px_v01_boston_seaport_unaug_gt_eval_preproc.tar.gz

Evaluate on partial observations: bev_nuscenes_256px_v01_boston_seaport_unaug_gt_eval_preproc.tar.gz

Evaluate on full observations: bev_nuscenes_256px_v01_boston_seaport_unaug_gt_full_eval_preproc.tar.gz

Training data

Download and extract the following compressed directories into the local data/ directory.

Note: The training datasets are 33 and 35 GB in size.

Data directory

dslp/
└───data/
    |   bev_nuscenes_256px_v01_boston_seaport_gt_full_preproc_train
    |   bev_nuscenes_256px_v01_boston_seaport_gt_preproc_train
    |   bev_nuscenes_256px_viz.tar.gz

Train on partial observations: bev_nuscenes_256px_v01_boston_seaport_gt_preproc_train.tar.gz

Train on full observations: bev_nuscenes_256px_v01_boston_seaport_gt_full_preproc_train.tar.gz

Static set of visualization samples used to monitor progress (required for running the code!): bev_nuscenes_256px_viz.tar.gz

Checkpoint files

Download checkpoint files into the local checkpoints/ directory.

Data directory

dslp/
└───checkpoints/
    |   ...

Experiment	NLL	IoU
exp_04_dslp_alpha_ib.ckpt	12.325	0.442
exp_08_dslp_region_1.ckpt	13.174	0.423
exp_09_dslp_region_1_2.ckpt	12.557	0.444

Evaluation

Run the evaluation script to recompute the main experiment results. The script assumes the datasets and checkpoints are set up as instructed.

sh run_eval_exp_04_dslp_alpha_ib.sh

Training

Run the training script to recreate the main experiment DSLP model. The script assumes the datasets and checkpoints are set up as instructed.

sh run_train_exp_04_dslp_alpha_ib.sh

Experimental results

Summary of results and baselines

Model	NLL	IoU
STSU (supervised)	-	0.389
LaneGraphNet (supervised)	-	0.420
DSLA	15.095	0.427
DSLP const alpha	12.663	0.418
DSLP mean alpha_ib	12.482	0.410
*DSLP alpha_ib	12.325	0.442
DSLP full obs.	12.205	0.454

'*' Our main result for partially observable worlds

All experiment quantitative result logs and output visualizations are uploaded.

Data directory

dslp/
└───results/
    └───exp_01_dsla/
    |   |   eval.txt <--Evaluation log
    |   |   results.txt <-- Evaluation summary
    |   |   viz_000.png <-- Output visualizations
    |   |   ...
    |
    └───exp_02_dslp_const_alpha/
    └───exp_03_dslp_mean_alpha_ib/
    └───exp_04_dslp_alpha_ib/ <-- Main result
    └───exp_05_dslp_full_obs/
    └───exp_06_dslp_no_world_model/
    └───exp_07_dslp_no_aug/
    └───exp_08_dslp_region_1/
    └───exp_09_dslp_region_1_2/