You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently we use python's built-in multiprocessing to spread computations across multiple CPUs. This requires that the metric function be globally importable, which is a limitation of pickle, which is used by multiprocessing. The package pathos offers a essentially drop-in replacement for multiprocessing which uses dill to serialize the objects that are passed to worker processes, which would allow for locally defined and more complex functions to be used as metrics. Currently this is not a major limitation, but if this issue crops up again in the future I think this would be an easy update to make.
Currently we use python's built-in
multiprocessing
to spread computations across multiple CPUs. This requires that the metric function be globally importable, which is a limitation ofpickle
, which is used bymultiprocessing
. The packagepathos
offers a essentially drop-in replacement formultiprocessing
which usesdill
to serialize the objects that are passed to worker processes, which would allow for locally defined and more complex functions to be used as metrics. Currently this is not a major limitation, but if this issue crops up again in the future I think this would be an easy update to make.Links with info:
https://github.com/uqfoundation/pathos
https://medium.com/@emlynoregan/serialising-all-the-functions-in-python-cd880a63b591
The text was updated successfully, but these errors were encountered: