Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement/switch multiprocessing to multithreading #216

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

tgrandje
Copy link
Collaborator

This is a suggestion to switch from multiprocessing to multithreading for the geodata package:

  • there are sometimes troubles using multiprocessing with windows machines, which are not easily debugged;
  • as far as I have seen, there is not much interest for multiprocessing as no heavy computation is done, and (I think) we're perfectly safe with multithreading.

This needed to rewrite the creation of the session object, which is now created outside the pool scope (instead of global objects created at threads/processes creation).

@tgrandje
Copy link
Collaborator Author

This is derived from this PR. As the geoparquet issue is still waiting for concensus, I chose to open a new PR working on this particular subject instead.

As already said here, the choice of using pebble is mostly lazyness. The advantage is that it's API si very similar to that of multiprocessing (instead of something like concurrent.futures for instead). I've never had any trouble installing pebble, so I think it is reliable enough for this...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant