use ThreadPool for multi-thread

[dixudx]

No description provided.

[dixudx]

cc @casabre @MartinBarkerPhilips

[casabre]

@dixudx I am wondering why did you revert the multiprocessing changes? I couldn't observe issues like in #151 but I had just around 90 issues compared to 302. Pagination should have worked because they were more than 50 issues.

Furthermore, I would recommend a ProcessPool rather than a ThreadPool for parsing the workitems because the GIL would limit real parallel computation here. I don't know if we loose any information regarding credentials etc. due to pickling but it worked for me with pathos and lambda functions.

[dixudx]

@casabre sorry for reverting your commits. When I added somes CIs for this project, the multiprocessing commits failed on many test cases. #151 (comment) also had such problems.

Furthermore, I would recommend a ProcessPool rather than a ThreadPool for parsing the workitems because the GIL would limit real parallel computation here. I don't know if we loose any information regarding credentials etc. due to pickling but it worked for me with pathos and lambda functions.

The pickling error is really a big headache. I've tried using ProcessPool, but it is not working.

Using pathos is fine, since it uses dill that can serialize any objects. But the fact is not like that, the unit tests keep failing.

[dixudx]

@casabre I've created another PR #155 using your commits. You can view the CI results against multiple Python versions.

_pickle.PicklingError: args[0] from __newobj__ args has the wrong class

Lots of pickling errors.

[casabre]

@dixudx I understand. Stability first! But I am really wondering why it was working out at my setup. I can offer that we can work out a solution together.
A multiprocess solution could be that we create the credential instance in the sub-process again by just forwarding the credential tuple string → less objects involved and easy serializing.

[casabre]

But you can stick at least with the multi-thread approach in base.py. There is just a ThreadPool involved which gains also a lot and even more in a sub-process 😉

[dixudx]

But you can stick at least with the multi-thread approach in base.py. There is just a ThreadPool involved which gains also a lot and even more in a sub-process

@casabre Sure. Please submit another PR.

I can offer that we can work out a solution together.
A multiprocess solution could be that we create the credential instance in the sub-process again by just forwarding the credential tuple string → less objects involved and easy serializing.

Yeah. Please feel free to work on this.

Using a top-level function is preferred instead of a sub-function.

[casabre]

@dixudx how do you format your code? I am using black as default and as far as I am seeing it, pycodestyle just checks for PEP compliance.

[dixudx]

Please try with

yapf --style google --recursive --in-place .

to format your codes.

[MartinBarkerPhilips]

cc @casabre @MartinBarkerPhilips

master does not work for be and results in an error (Caused by SSLError(SSLError(1, '[SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:997)'))) because I am running an older version of RTC I believe

[dixudx]

@MartinBarkerPhilips What is your Python version?

[MartinBarkerPhilips]

@MartinBarkerPhilips What is your Python version?

I am on Python 3.10.4