FuturePool is a package that introduce known concept of multiprocessing Pool to the async/await world, resulting in async workers pool library. It allows for easy translation from multiprocessing to async/await, while keeping the core principle - specified number of workers. FuturePool allows for more flexible usage by providing starimap/starimap_unordered.
FuturePool was created to handle web scrapping, where in order to not overwhelm website with connections and comply with website requirements, specified number of workers was used. FuturePool was extended to handle generic scenarios and published.
MIT
To see library docs visit https://MichalKarol.github.io/futurepool/.
Example translation from multiprocessing to FuturePool
# multiprocessing
from multiprocessing import Pool
from time import sleep
def pool_fn(i):
sleep(i)
return i
with Pool(2) as p:
result = p.map(pool_fn, range(10))# FuturePool
from futurepool import FuturePool
from asyncio import sleep
async def async_pool_fn(i):
await sleep(i)
return i
async with FuturePool(2) as fp:
result = await fp.map(async_pool_fn, range(10))Michał Karol [email protected]
Mastodon
Github