Evaluate current Localization accuracy of current NDT scan matching against Localization dataset

[mitsudome-r]

Description

We should compare the localization performance of Autoware before/after Autoware Labs activity.
Once we create a pointcloud map for the data collected in this issue, we should do evaluation of current Autoware's localization accuracy.

Purpose

This issue aims to measure the localization performance of autoware in urban environment.

Possible approaches

For the accuracy, we can try using post-processed GNSS/INS data output as ground truth.

Definition of done

1. Displaying localization errors with metrics, using Ground Truth (Post Processed GNSS/INS data) as reference..
2. Determining where localization errors increase in urban environment and where existing autoware localization is inadequate.

[mitsudome-r]

There is no guarantee that RTK is always accurate so I appreciate for any other way of getting ground truth data.

[meliketanrikulu]

I tested with data which is this PR shared.
I tested all map with default autoware. I used GNSS/INS Twist Messages instead vehicle velocity because this data collected with mapping setup and there is no vehicle velocity information on it.

1.) Here is the localization broken points in here on the google earth.
You can open this .kml file on google earth.

2.) Also you can see broken points on pointcloud map in here
You can open this file with Cloud Compare.
I will also share test video.

[meliketanrikulu]

I made some parametric changes and also we collected new data because of time problem in last data. You can check out the some test results with new data.
1.) To start, I simply wanted to observe how autoware localization works. Test Video

I did this test with Default autoware. Since I don't have the vehicle velocity as a Twist source, I did this test using the TwistWithCovarianceStamped message from GNSS/INS.
There are 2 tunnels on our route. As expected, localization was disrupted while passing through the tunnels. Apart from this, no visible disruption was observed in localization at first glance.

2.) The error of GNSS/INS in the tunnel increases as seen in the video above.However, the frequency of the Twist message does not decrease or the Twist message is not interrupted. For this reason, I wanted to check if using the speed coming from here causes an error in the tunnel. For this reason, I tested the localization in the tunnel by giving only NDT poses as input to EKF. In this test, no Twist message was given to EKF. Test Video

[meliketanrikulu]

I added Show&Tell Discussion for explaining to tests --> https://github.com/orgs/autowarefoundation/discussions/5135

[meliketanrikulu]

Added document to Autoware Documentation . We can close this issue .

[Cyohou]

You said, “For the accuracy, we can try using post-processed GNSS/INS data output as ground truth.”

Regarding this statement, I would like to ask, “How is the post-processed GNSS/INS data set up, and how are the GNSS/INS coordinates converted into comparable global coordinates?”