Model performance on different types of input

[shrey-roongta]

Hi, your project is very interesting. For my ML course, I have to make a small contribution to an open source ML project. My current idea is to analyze your model's performance on different types of input. However, I'm finding it difficult to automate the testing process (I'm trying to use eval.py). My major concern is that if I scrape random images from the web, how should I get their true label? I also tried to use a .pkl file found on Kaggle but I get the following error

Please let me know if you have any ideas on this. Thank you for your amazing work!

[lukas-blecher]

Hi, thanks for your interest in the project.
What do you mean by "different types of input"? I do have some scraping tools in the dataset directory but they are not perfect. The regex I am using to extract the true labels from the website source code still give false positives but I didn't have time to look into it yet.
Which sites do you want to scrape? If they have rendered out images it is likely that the true label is not available. Many of the potential websites that do have Latex equations render them using Mathjax or KaTeX. That's how I got some training data from wikipedia and the likes. However it is often not available in image format. Wikipedia is different, as there is also a svg image associated to the latex code which can easily be converted to a png using image magick or similar. So you can look into my scraping code and adapt it.
But I'm not sure what your end goal is with this. Do you want to compile a validation set that aims to represent the real world data distribution better?

What kaggle pkl file do you mean? It's most likely not compatible out of the box. You will have to create your own pkl file.