Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed a bug affecting table mapping in evaluate_result_str() #1

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

mnamysl
Copy link

@mnamysl mnamysl commented May 8, 2022

This PR fixes an issue affecting table matching in the evaluate_result_str() function. In particular, the elements of a sequence were removed when iterating over the sequence, resulting in some elements being skipped and matching being incomplete.

To validate that, I ran the evaluation of the ground-truth data against itself, hoping to get the perfect scores. I used the annotations for the trackB2 (modern documents subset) in the ICDAR 2019 cTDaR table competition. I got the following results:

IOU @ 0.6 -
precision: 0.7933181856433561
recall: 0.7933181856433561
f1: 0.7933181856433561
correct: 9427, gt: 11883, res: 11883
IOU @ 0.7 -
precision: 0.7933181856433561
recall: 0.7933181856433561
f1: 0.7933181856433561
correct: 9427, gt: 11883, res: 11883

IOU @ 0.8 -
precision: 0.7933181856433561
recall: 0.7933181856433561
f1: 0.7933181856433561
correct: 9427, gt: 11883, res: 11883
IOU @ 0.9 -
precision: 0.7933181856433561
recall: 0.7933181856433561
f1: 0.7933181856433561
correct: 9427, gt: 11883, res: 11883

After applying the suggested fix, the expected result is returned.

Evaluation of trackB2
IOU @ 0.6 -
precision: 1.0
recall: 1.0
f1: 1.0
correct: 11883, gt: 11883, res: 11883
IOU @ 0.7 -
precision: 1.0
recall: 1.0
f1: 1.0
correct: 11883, gt: 11883, res: 11883

IOU @ 0.8 -
precision: 1.0
recall: 1.0
f1: 1.0
correct: 11883, gt: 11883, res: 11883
IOU @ 0.9 -
precision: 1.0
recall: 1.0
f1: 1.0
correct: 11883, gt: 11883, res: 11883

@mnamysl mnamysl changed the title fixed a bug affecting table mapping in evaluate_result_str() Fixed a bug affecting table mapping in evaluate_result_str() May 8, 2022
Copy link

@alexander-esser alexander-esser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Used enumerate in the for-loop; in order to know the index within the loop and to remove the erroneous line remaining_tables.remove(rest) # unsafe... should be ok with the break below

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants