Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Significant increase in number of tasks of test_climatology::test_highlevel_api #1622

Open
fjetter opened this issue Dec 2, 2024 · 1 comment

Comments

@fjetter
Copy link
Member

fjetter commented Dec 2, 2024

I just noticed that dask/dask#11479 multiplied the number of tasks by 2.5x before and by 4x after optimization. I haven't looked at what this does to the end to end runtime but initial submission and client/scheduler side memory obviously takes a hit. Given that before/after optimization task number ratio got much worse I suspect we're loosing opportunities for trivial fusion due to this rechunking

# before https://github.com/dask/dask/pull/11479
Enter optimize 210477
After optimize_blockwise 166163
After fuse_roots 166163
After cull 77002
Low level 77002
After fuse 71878
After inline 71878
After opt slices 71878

# after https://github.com/dask/dask/pull/11479
Enter optimize 535113
After optimize_blockwise 390839
After fuse_roots 390839
After cull 301678
Low level 301678
After fuse 293482
After inline 293482
After opt slices 293482
@hendrikmakait
Copy link
Member

I think this is expected. We should probably compare the current task count to something like the workload before #1583.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants