Adding multiprocessing

[casabre]

First implementation of multiprocessing for review

[casabre]

@dixudx which formatting tool are you using? I would adapt my changes to this because my VS Code formatted with Black...

[casabre]

I observed a speed-up of approximately 43% for a runSavedQueryByID call with 82 entries. However, the parsing takes quite long time in comparison to the requests.get round-trip time.

• Time with old implementation for the described call ~145 seconds
• Time with multiprocessing for the described call ~83 seconds
• The requests.get call takes ~1.5 seconds

[dixudx]

Looks good to me.
Would you please format the code? Thanks

[dixudx]

which formatting tool are you using? I would adapt my changes to this because my VS Code formatted with Black...

@casabre Please run tox -e pycodestyle. Refer to the testing guide for details.

[dixudx]

I observed a speed-up of approximately 43% for a runSavedQueryByID call with 82 entries. However, the parsing takes quite long time in comparison to the requests.get round-trip time.

* Time with old implementation for the described call `~145 seconds`

* Time with multiprocessing for the described call `~83 seconds`

* The `requests.get` call takes `~1.5 seconds`

Really HUGE improvements. 👍🏻👍🏻👍🏻

I am wondering whether we could have a benchmark graph on our README.

[casabre]

Really HUGE improvements. 👍🏻👍🏻👍🏻

Thanks a lot but I dug deeper because I was wondering about the long overall processing time. I re-used the previous mentioned scenario. There are now 83 entries in the query.

Action	Processing Time	Comment
requests.get	~ 1.5 - 2 seconds	Bias via VPN
xmltodict.parse	~ 0.1 seconds
process query dict	~ 85 - 90 seconds	Process spawning takes ~ 5 seconds. Running it in a row uses the instance from garbage collection which speeds up

I am wondering where the processing time is lost because I used 8 cores which should at least speed up by factor 8 and not ~2... just breaking down the result of the old implementation into single times, I am ending up with ~2 seconds per work item. Considering the 8 processes with constant processing time of ~2 seconds, I would assume at least 20 seconds overall processing time ± some processing scheduling overhead.

I don't know any detail about the underlying implementation but is any work item queried separately in the mapping phase? → #149 could help a lot even without multiprocessing because we can parse during the await request.get phase.

I am wondering whether we could have a benchmark graph on our README.

Sure, we can do. I can prepare a Jupyter notebook. Do you have a reasonable setup? Currently, I am working from home via a VPN connection which biases the results 😉

[casabre]

Furthermore, I don't know if json parsing is faster compared to xml parsing. With ujson there is a least a fast json implementation available.
No significant improvements with json

[casabre]

I found also a spot in the lowest base.py layer which was looking for a MP change 😉. Now, the speed improvements are going from factor ~2 to ~6.

Change	Overall time
Initial	~145 seconds
ProcessPool for _get_paged_resources	~80 - 90 seconds
ProcessPool for _get_paged_resources + ThreadPool for __initializeFromRaw	~24 seconds

I was wondering why looping over the OrderedDict items is taking that long. There is no complex data filtering/mapping applied.

[dixudx]

Looks good to me. Thanks for such a HUGE improvement.

[casabre]

@dixudx when are you planning a new release? Would need quite soon because Git cloning won't become an option in future :).

[dixudx]

@casabre Sure. Already shipped with 0.8.0. You can install with pip now.