Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

postprocessing/subjob example needed #960

Open
chrisdane opened this issue May 21, 2023 · 3 comments · May be fixed by #1057
Open

postprocessing/subjob example needed #960

chrisdane opened this issue May 21, 2023 · 3 comments · May be fixed by #1057
Assignees
Labels
documentation Improvements or additions to documentation workflow manager

Comments

@chrisdane
Copy link
Contributor

Request/contribution description
I am confused by the documentation on how to set up post-processing jobs/subjobs as e.g. here: https://esm-tools.readthedocs.io/en/latest/workflow.html?highlight=tidy#example-1-adding-an-additional-postprocessing-subjob:

echam:
        [...other information...]

        workflow:
                next_run_triggered_by: tidy

                subjobs:
                        my_new_subjob:
                                nproc: 1
                                run_after: tidy
                                script_dir:
                                script:
                                call_function:
                                env_preparation:

My question are:

  • why is this in a specific model section (here echam)?
  • are both next_run_triggered_by: tidy and run_after: tidy necessary? Whats their difference?
  • same for script_dir and script; why both?
  • what are call_function and env_preparation?
  • can't nproc be provided by e.g. a usual sbatch-header in the script itself?

I would find an explicit example very helpful.

Thanks a lot and cheers,
Chris

@chrisdane chrisdane added the documentation Improvements or additions to documentation label May 21, 2023
@nwieters
Copy link
Contributor

Hi @chrisdane,

as a first reply to your issue.

I followed the exercise in the ESM-Tools workshop (https://github.com/esm-tools/workshops/blob/main/202207_AWI_ESM-Tools/exercises/7.5_workflow_manager.md see the solution).

And this worked for me as a first try:

flow:
    workflow:
        next_run_triggered_by: tidy
 
        subjobs:
            my_new_subjob:
                batch_or_shell: batch
                order_in_cluster: concurrent
                run_on_queue: ${computer.partitions.pp.name}
                nproc: 1
                run_after: tidy
                script_dir: "/work/ab0995/a270089/myrunscripts/"
                script: "hallowelt.sh"
                submit_to_batch_system: True

I placed this in a runscript as a new component. What also worked is to put everything starting from workflow in another component, as you mentioned in your issue. I also do not understand why this needs to be in a component.
I also had the same questions about the keywords as you mentioned. I agree that this needs to be improved in the documentation with some more examples.

If you have any other suggestions for the workflow manager, please also add them as a comment to the following discussion #1023. Thank you.

@nwieters nwieters self-assigned this Sep 27, 2023
@nwieters nwieters linked a pull request Sep 27, 2023 that will close this issue
Copy link

This issue has been inactive for the last 365 days. It will now be marked as stale and closed after 30 days of further inactivity. Please add a comment to reset this automatic closing of this issue or close it if solved.

@github-actions github-actions bot added the Stale label Sep 26, 2024
@mandresm
Copy link
Contributor

This issue cannot yet be closed.

I also do not understand why this needs to be in a component.

Maybe we can change this in the future, so that it doesn't have to be in a component. More on that when we resume the work on the workflow manager.

@github-actions github-actions bot removed the Stale label Sep 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation workflow manager
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants