Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up wiki scraping #42

Merged
merged 2 commits into from
Feb 15, 2024
Merged

Speed up wiki scraping #42

merged 2 commits into from
Feb 15, 2024

Conversation

robotboy655
Copy link
Contributor

This is different from #31, this PR aims to speed up the "initial" wiki scrape. It is way too slow right now.

Also wipe the output folder by default when running npm run scrape-wiki, so that files do not have duplicated data.

This significantly speeds up wiki scraping
Is there a reason not to do this?
@luttje
Copy link
Owner

luttje commented Feb 15, 2024

Hi there, thanks a bunch for contributing to this little project. I really appreciate you taking time form working on Garry's Mod to help out here.

IIRC me not wiping the output folder by default was so I could separately test the packing functionality on preexisting output (without accidentally wiping my "test data"). However since everything seem to work alright, there's no reason not to wipe now.

The scraping really is a lot faster thanks to you. I remember thinking taking the slow way would be better not to trigger any throttling on the wiki server. You coming in with this PR really helps, since I wouldn't have dared to make such a change myself.

Thanks again!

@luttje luttje merged commit 5213c35 into luttje:main Feb 15, 2024
1 of 2 checks passed
@luttje
Copy link
Owner

luttje commented Feb 15, 2024

@all-contributors add @robotboy655 for code

Copy link
Contributor

@luttje

I've put up a pull request to add @robotboy655! 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants