Why do I get negative MOTA?

[SamGhK]

Hello,

many thanks for your work and making your code public. Are my results correct? I have not change anything. I have cloned the repository, put the data under right folders and executed "pp_pv_40e_mul_C/eval.sh"

I look forward to hearing from you.

Best regards

==========================tracking evaluation summary===========================
Multiple Object Tracking Accuracy (MOTA) -0.044093
Multiple Object Tracking Precision (MOTP) 0.854040
Multiple Object Tracking Accuracy (MOTAL) 0.789259
Multiple Object Detection Accuracy (MODA) 0.789616
Multiple Object Detection Precision (MODP) 0.890653

Recall 0.891036
Precision 0.922639
F1 0.906562
False Alarm Rate 0.236567

Mostly Tracked 0.773148
Partly Tracked 0.203704
Mostly Lost 0.023148

True Positives 11342
Ignored True Positives 1616
False Positives 951
False Negatives 1387
Ignored False Negatives 1192
Missed Targets 1387
ID-switches 9265
Fragmentations 9295

Ground Truth Objects (Total) 13921
Ignored Ground Truth Objects 2808
Ground Truth Trajectories 240

Tracker Objects (Total) 13168
Ignored Tracker Objects 875
Tracker Trajectories 13130

[ZwwWayne]

Hi @SamGhK ,
Thanks for your feedback, this is unusual because the metrics except the MOTA seems to be normal.
Can you provide more details about your running environment? Especially the version of motmetrics etc. It would do some help to debug.

[SamGhK]

Many thanks for your fast response. The running environment is as follow:
I am not using conda environment and srun. The required packages are install via:
pip3 install -r requirements.txt
pip3 install numba scikit-image scipy pillow
OS: Ubuntu 18.04 LTS
Python: 3.6.9
torch: 1.3.1 (built from source)
torchvision: 0.4.2
cuda: 10.2
cuDNN: 7.6.5
motmetrics: 1.1.3
cython: 0.29.14
numba: 0.47.0
opencv: 3.4.8
pyproj: 2.4.2.post1
ortools:7.4.7247
munkres: 1.1.2

Looking forward to hearing from you.

Best regards

[ZwwWayne]

Hi @SamGhK ,
I evaluated the model again and got the same normal results as before. I thought the problem is the version of pyproj, because I met the problem on pyproj before the code release, but now I am using the same pyproj package as you do.
For now, I suggest you follow the conda installation process and try again, the conda installation should guarantee success. In the meantime, I will also try to find out the problem.

[SamGhK]

Hello @ZwwWayne,

Thanks for your support. Although I don't like conda, but I will follow the conda installation. I have other questions. how can I force the model to not use the LiDar module or force it to use only 2D images? I am trying to replace the VGG-Net with a Densnet for better 2D-BB estimation. Is it possible to do that?I have seen that the model is capable or different ResNets, so I had the Idea that it should be possible to replace the VGG Net. Thanks in advance.

Looking forward to hearing from you.

Best regards

[ZwwWayne]

It is possible to do that.
You need to set point_len: 512 to point_len: 0 to avoid using LiDAR modules.
And then you need to set score_fusion_arch: ~, test_mode: 0 to avoid use fusion module.
You might also need to check the score_fusion_arch to set fusion_module in this line.

[ZwwWayne]

Hi @SamGhK ,
The eval results of pp_pv_40e_mul_C should be as the following:

You can check it with your evaluation results.

[SamGhK]

Hi,
I checked your model with conda env.

Conda installation:
curl -o ~/miniconda.sh -O https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh && \ chmod +x ~/miniconda.sh && \ ~/miniconda.sh -b -p /opt/conda && \ rm ~/miniconda.sh && \

**Dependencies: **
conda update -n base -c defaults conda && \ conda install pytorch torchvision -c pytorch && \ conda install numba cython && \ conda install -c conda-forge opencv && \ pip install motmetrics PyYAML easydict munkres ortools pyproj && pip uninstall -y pillow && pip install pillow==5.2.0

I am not getting any negative results as follow:
Eval after training:

Eval:

But my MOTA result does not match yours.
I execute experiments/pp_pv_40e_mul_C/eval.sh and have noticed the following point at the start of process:

=> no checkpoint found at '.data/pretrain_models/pp_pv_40e_mul_C-gpu.pth'
Building dataset using dets file ./data/pretrain_models/pp_train_dets.pkl
Detect [ 16258] cars in [3365/3975] images
Add [0] cars in [0/3975] images
Building dataset using dets file ./data/pretrain_models/pp_val_dets.pkl
Detect [ 13170] cars in [3475/3945] images
Add [0] cars in [0/3945] images
[2020-01-13 17:33:34,592][eval_seq.py][line: 64][ INFO] args: Namespace(config='experiments/pp_pv_40e_mul_C/config.yaml', evaluate=False, load_path='.data/pretrain_models/pp_pv_40e_mul_C-gpu.pth', memory=False, recover=False, result_path='experiments/pp_pv_40e_mul_C/results', result_sha='all')
[2020-01-13 17:33:34,593][eval_seq.py][line: 65][ INFO] config: {'augmentation': {'input_size': 224, 'test_resize': 224},

Do you have any suggestion, why am I getting worse results?

Best regards and thanks in advance,
Sam

[SamGhK]

Hi @ZwwWayne ,
I have an update on my problem. I debugged my env and landed in box_np_ops.py @line 725, function
def corner_to_surfaces_3d_jit(corners):

I have changed
@numba.njit(nopython=True) to @numba.njit(parallel=True, fastmath=True)

Now, in my original env (no conda) I am getting positive results, but still can't get your result.

But I can't figure it out what is no functioning correctly. I would appreciate if you could give me a feedback/suggestion.

Look forward to it.

Best regards,
Sam

[ZwwWayne]

=> no checkpoint found at '.data/pretrain_models/pp_pv_40e_mul_C-gpu.pth in the log means the model is not loaded successfully.
You might need to check whether the path of the pre-trained model is correct.

[SamGhK]

Hi @ZwwWayne ,

many thanks for your feedback. You were right. it was because this line.

mmMOT/experiments/pp_pv_40e_mul_C/eval.sh

Line 5 in 542f581

--load-path=./pretrain_models/pp_pv_40e_mul_C-gpu.pth \

I removed the point from the beginning of the pretrain_models - path and it worked. Now, I get the same result like yours. I guess the problem was

1. Not being able to load the pretrained model
2. the Numba.jit nopython mode @numba.jit(nopython=True) @ following line
   

   mmMOT/point_cloud/box_np_ops.py

   Line 723 in 542f581

   @numba.jit(nopython=True)

.
I became suspicious after I decided to train the model myself and got the following results during training:

But then it got frustrating again because my training was interrupted because of the following error:

Do you have any idea, what the problem could be, that interrupted the training @ 87200th iteration?

I look forward to hearing from you.

Best regards,
Sam

[ZwwWayne]

1. The evaluation at the iteration of 83875 seems to be fine.
2. It seems the id_offset and dets_out[i]['id'] have different types, you can print them out and check what happened.

[simaker]

It is possible to do that.
You need to set point_len: 512 to point_len: 0 to avoid using LiDAR modules.
And then you need to set score_fusion_arch: ~, test_mode: 0 to avoid use fusion module.
You might also need to check the score_fusion_arch to set fusion_module in this line.

It is possible to do that.
You need to set point_len: 512 to point_len: 0 to avoid using LiDAR modules.
And then you need to set score_fusion_arch: ~, test_mode: 0 to avoid use fusion module.
You might also need to check the score_fusion_arch to set fusion_module in this line.

and which steps are necessary to use LiDAR only?

[ZwwWayne]

Keep point_len: 512 and set appear_len: 512 to appear_len: 0.
And then set score_fusion_arch: ~, test_mode: 0 to avoid use fusion module.

[simaker]

but why do i have to set test_mode: 0 and not 1?
in the config it says:
test_mode: 1 #0:image;1:LiDAR;2:fusion

[ZwwWayne]

Because when only using a single modality, the batch size dimension is always 1.
Using 0, 1, 2 for different modalities is only applicable to fusion methods.

[simaker]

okay thank you. And how do i have to change this line? I don't know what you mean by

check the score_fusion_arch to set fusion_module

[llllliu1]

Many thanks for your fast response. The running environment is as follow:
I am not using conda environment and srun. The required packages are install via:
pip3 install -r requirements.txt
pip3 install numba scikit-image scipy pillow
OS: Ubuntu 18.04 LTS
Python: 3.6.9
torch: 1.3.1 (built from source)
torchvision: 0.4.2
cuda: 10.2
cuDNN: 7.6.5
motmetrics: 1.1.3
cython: 0.29.14
numba: 0.47.0
opencv: 3.4.8
pyproj: 2.4.2.post1
ortools:7.4.7247
munkres: 1.1.2

Looking forward to hearing from you.

Best regards
Hello, excuse me. I'm not clear about what directories are in kitti_t_o/, could you please tell me,thank you!

[ghost]

You need to download all the data and unzip it from the KITTI Tracking Benchmark

Download left color images of tracking data set (15 GB) and unzip it to 'image_02' folder
Download Velodyne point clouds, if you want to use laser information (35 GB) and unzip it to 'velodyne' folder
Download GPS/IMU data, if you want to use map information (8 MB) and unzip it to 'oxts' folder
Download camera calibration matrices of tracking data set (1 MB) and unzip it to 'calib' folder
Download training labels of tracking data set (9 MB) and unzip it to 'label_02' folder
Download L-SVM reference detections for training and unzip it to 'lsvm' folder
Download Regionlet reference detections for training and unzip it to 'regionlets' folder
Download tracking development kit (1 MB) and unzip it to 'devkit' folder