Simplify parallel backend config #549

AnesBenmerzoug · 2024-03-27T20:22:06Z

Description

This PR closes #385

Changes

Deprecated config argument of parallel backend classes, ray future executor and MapReduceJob class.
Allowed instantiating parallel backend classes directly and deprecated init_parallel_backend function.
Updated docstrings.
Adapted all methods to use parallel_backend and deprecate config argument.

Checklist

Wrote Unit tests (if necessary)
Updated Documentation (if necessary)
Updated Changelog
~~If notebooks were added/changed, added boilerplate cells are tagged with "tags": ["hide"] or "tags": ["hide-input"]~~

…ig fixture

This should ideally be done better. Perhaps we could wrap this pattern similarity to the map reduce pattern and pass it there

mdbenito

As usual, good stuff. I left some low-value/interest comments, and one about documentation which I do think is important.

src/pydvl/parallel/backends/joblib.py

src/pydvl/parallel/backend.py

src/pydvl/parallel/backends/joblib.py

src/pydvl/parallel/map_reduce.py

docs/getting-started/advanced-usage.md

src/pydvl/parallel/backends/ray.py

tests/parallel/test_parallel.py

kosmitive · 2024-04-02T09:11:32Z

Really cool! I would also vote for making the whole parallel module a separate library it is really useful as a standalone package.

…d use it in MapReduceJob

mdbenito

Left some changes. I will apply them and merge

docs/getting-started/advanced-usage.md

src/pydvl/parallel/__init__.py

mdbenito

Good schtuff

AnesBenmerzoug added 14 commits March 27, 2024 13:48

Allow instantiating parallel backend classes directly

d7143be

Deprecate use of config argument in JoblibParallelBackend

75dbca0

Deprecate use of config argument in RayParallelBackend

cf90870

Deprecate use of config argument in RayExecutor

96b7ec9

Deprecated use of config argument in RayExecutor

dfeeadd

Rename BaseParallelBackend to ParallelBackend

01f2c2c

Deprecate use of config argument for future executor classes

6597a9e

Deprecate use of init_executor and config in compute_loo

bf12873

Docstring changes

3111533

Make maybe_init_parallel_backend a private function

5a09aa4

Improve type annotations

2a64ad4

Deprecate use of config in MapReduceJob

e640eac

Fixes

2f66acf

Create new parallel backend fixture and use it in loo test

c570880

AnesBenmerzoug self-assigned this Mar 27, 2024

AnesBenmerzoug added 15 commits March 29, 2024 16:47

Deprecate use of config in montecarlo least core

31683fb

Deprecate use of config in combinatorial exact shapley

77297ac

Deprecate use of config in montecarlo shapley

f32455b

Deprecate use of config in owen shapley

ff57e55

Deprecate use of config in classwise shapley

0d3c26d

Deprecate use of config in group testing shapley

993e6ee

Update docstrings

866e754

Deprecate use of config in semivalues

d5ae8ea

Remove effective_n_jobs function

acfc3e6

Update tests to use parallel_backend fixture instead of parallel_conf…

7108087

…ig fixture

Hardcode wait timeout in permutation_montecarlo_shapley

d905fd2

This should ideally be done better. Perhaps we could wrap this pattern similarity to the map reduce pattern and pass it there

Ignore return type of functions wrapped with @deprecated

71b0a47

Fix arguments to joblib's Parallel

bee00fa

Deprecate use of config in lc_solve_problems

59230f6

Update parallel backend documentation

832cd3a

AnesBenmerzoug added 5 commits March 29, 2024 18:28

Replace parallel_config fixture

4b4a258

Fix tests

fd25ce7

Fix effective_n_jobs methods in joblib parallel backend

67342d3

Fix tests

f5c6ee0

Disable mkdocs social plugin

31dcbc9

AnesBenmerzoug requested review from mdbenito and schroedk March 30, 2024 19:36

Update changelog

863bc39

AnesBenmerzoug marked this pull request as ready for review April 1, 2024 11:47

mdbenito reviewed Apr 1, 2024

View reviewed changes

Use cast instead of colon syntax for joblib executor

a6d9633

kosmitive reviewed Apr 2, 2024

View reviewed changes

src/pydvl/parallel/backends/ray.py Show resolved Hide resolved

tests/parallel/test_parallel.py Show resolved Hide resolved

AnesBenmerzoug added 8 commits April 2, 2024 18:35

Use new union and optional syntax for type annotations

270038a

Only allow use of JoblibParallelBackend with MapReduceJob

324143f

Deprecate only config argument of init_parallel_backend

6a7313c

Define _joblib_backend_name attribute for parallel backend classes an…

55e06af

…d use it in MapReduceJob

Document mapreducejob and futures executor interfaces

65313e7

Merge branch 'develop' into feature/simplify-parallel-backend-config

371ef79

Reenable the mkdocs social plugin

b0a7a49

Add instructions about init_parallel_backend

e23256b

AnesBenmerzoug requested a review from mdbenito April 5, 2024 09:08

mdbenito reviewed Apr 5, 2024

View reviewed changes

Minor tweaks

61f69c1

mdbenito approved these changes Apr 5, 2024

View reviewed changes

mdbenito merged commit d40e8e2 into develop Apr 5, 2024
0 of 5 checks passed

mdbenito deleted the feature/simplify-parallel-backend-config branch April 5, 2024 09:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simplify parallel backend config #549

Simplify parallel backend config #549

AnesBenmerzoug commented Mar 27, 2024 •

edited

Loading

mdbenito left a comment

kosmitive commented Apr 2, 2024 •

edited

Loading

mdbenito left a comment

mdbenito left a comment

Simplify parallel backend config #549

Simplify parallel backend config #549

Conversation

AnesBenmerzoug commented Mar 27, 2024 • edited Loading

Description

Changes

Checklist

mdbenito left a comment

Choose a reason for hiding this comment

kosmitive commented Apr 2, 2024 • edited Loading

mdbenito left a comment

Choose a reason for hiding this comment

mdbenito left a comment

Choose a reason for hiding this comment

AnesBenmerzoug commented Mar 27, 2024 •

edited

Loading

kosmitive commented Apr 2, 2024 •

edited

Loading