Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace multiprocessing with pathos #7

Open
agartland opened this issue Jul 28, 2020 · 0 comments
Open

Replace multiprocessing with pathos #7

agartland opened this issue Jul 28, 2020 · 0 comments

Comments

@agartland
Copy link
Owner

Currently we use python's built-in multiprocessing to spread computations across multiple CPUs. This requires that the metric function be globally importable, which is a limitation of pickle, which is used by multiprocessing. The package pathos offers a essentially drop-in replacement for multiprocessing which uses dill to serialize the objects that are passed to worker processes, which would allow for locally defined and more complex functions to be used as metrics. Currently this is not a major limitation, but if this issue crops up again in the future I think this would be an easy update to make.

Links with info:
https://github.com/uqfoundation/pathos
https://medium.com/@emlynoregan/serialising-all-the-functions-in-python-cd880a63b591

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant