Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

benchmark needed #8

Open
lwb2099 opened this issue Dec 13, 2023 · 3 comments
Open

benchmark needed #8

lwb2099 opened this issue Dec 13, 2023 · 3 comments

Comments

@lwb2099
Copy link

lwb2099 commented Dec 13, 2023

No description provided.

@lwb2099
Copy link
Author

lwb2099 commented Dec 13, 2023

are results tested without glyph instructions? also it better if you can provide eval dataset rather than prompts template for better reproduction

@yukang123
Copy link
Collaborator

yukang123 commented Dec 20, 2023 •
edited
Loading

Hi, thanks for reaching out! For OCR accuracy and Clip Score, we use the SimpleBench and CreativeBench. The glyph instructions are set as default without deliberate design (following the inference codes). We will release the benchmark testing codes this week. Sorry for the delay release.

While for the FID score testing, we use 10K samples which are not used for training in LAION-Glyph for evaluation. Hopefully we will release this eval set in a week or so.

@yukang123
Copy link
Collaborator

The full prompts of two eval benchmarks, which are related to the OCR results shown in the paper, could be found in the text_prompts/paper folder.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yukang123 @lwb2099