Update make_ww3_*_file workers and their place in the automation workflow
#213
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Enables wwatch3 to be run on different VMs to those where NEMO is run. This is a step towards running wwatch3 nowcast/forecast runs concurrently with nowcast-green.
douglatornell
added
documentation
douglatornell
force-pushed
the
update_ww3-runs
branch
from
5839684 to
96f8cc4
Compare
Add `chunks` and `drop_variables` parameters into `xarray.open_mfdataset()` in both workers and tests to optimize data loading and memory usage. This update improves efficiency and performance by limiting loading of unnecessary variables and enables parallel processing for large datasets. The hope is that these changes will resolve (or at least reduce the frequency of) the issue of one or both of the `make_ww3*` workers stalling during the `ds.to_netcdf()` step in which dask is computing the task graph.
douglatornell
changed the title
make_ww3_*_file workers and their place in the automation workflow
Always launch make_ww3_*_file workers after NEMO forecast finishes instead of using a config flag to control whether they launch then or after nowcast-green runs. Concurrency is possible because we have changed to running wwatch3 on different VMs to NEMO.
|
Launch of |
|
Stalling of one or both of the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add mpi hosts file to wave forecasts configuration
Enables wwatch3 to be run on different VMs to those where NEMO is run. This is a step towards running wwatch3
nowcast/forecast runs concurrently with nowcast-green.
Add
chunksanddrop_variablesparameters intoxarray.open_mfdataset()callsThe hope is that these changes will resolve (or at least reduce the frequency of) the issue of one or both of the
make_ww3*workers stalling during theds.to_netcdf()step in which dask is computing the task graph.Move
make_ww3_*_fileworker launches to after NEMO forecast run so that wwatch3 nowcast and forecast runs are concurrent with NEMO nowcast-green run