Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Turn off jb eval #9

Merged
merged 3 commits into from
Dec 9, 2024
Merged

Turn off jb eval #9

merged 3 commits into from
Dec 9, 2024

Conversation

guillaumevernieres
Copy link
Contributor

I'm not 100% sure this fixes the memory issue on MSU HPC, but, while I was working on other things, I turned off the jb eval out of impatience to gain ~5+mn of runtime. I also noticed that the memory footprint is down to ~0.8TB for the 3DVARFGAT.
I'm testing with more iterations right now.

Copy link
Collaborator

@danholdaway danholdaway left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this @guillaumevernieres. For backward compatibility I think it would be good to default to True. And also allow users the option to keep using the feature. Then you can set the two settings to False in your jcb settings.

3dfgat.yaml.j2 Outdated Show resolved Hide resolved
3dvar.yaml.j2 Outdated Show resolved Hide resolved
3dfgat.yaml.j2 Outdated Show resolved Hide resolved
3dvar.yaml.j2 Outdated Show resolved Hide resolved
@danholdaway
Copy link
Collaborator

danholdaway commented Dec 9, 2024

By the way, when you create a jcb-algorithms PR, you also need to create an empty branch with the same name in the main jcb repo. This is unfortunately needed for the CI. It allows for the connection between jcb-algorithms and repos like jcb-gdas so that these changes can be tested against those repos. I'll try to remove that requirement soon.

@guillaumevernieres
Copy link
Contributor Author

Roger that @danholdaway ...

It looks like this is way more than a 5 mn gain.

  • 30 iterations wo j eval : 17mn
  • " " with : probably 30+mn. I's still running but I'm pretty sure it will time out (only asked for 30mn on debug)

We should turn that option off by default, this has been a huge waste of resources.

@danholdaway
Copy link
Collaborator

I'm OK with setting the flags to e.g. jb evaluation: {{ jb_evaluation | default(false) }}

@guillaumevernieres guillaumevernieres marked this pull request as ready for review December 9, 2024 20:35
@danholdaway danholdaway merged commit f25c1e5 into develop Dec 9, 2024
1 check passed
@danholdaway danholdaway deleted the bugfix/nojb branch December 9, 2024 20:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants