Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Conveniences for developing and using runfiles #32

Open
nbren12 opened this issue Jan 29, 2020 · 0 comments
Open

Conveniences for developing and using runfiles #32

nbren12 opened this issue Jan 29, 2020 · 0 comments

Comments

@nbren12
Copy link
Contributor

nbren12 commented Jan 29, 2020 •
edited
Loading

These occured to me while implementing the prognostic runs. I think these things should be enalbed on the python level

  1. Easy access to the configuration object. (e.g. add a get_config function)
  2. Easy logging for parallel systems
  3. Add logging to internal fv3gfs functions.
  4. add runfile, image, etc. to the config object. This is related to some work on reproducibility that @oliverwm1 is exploring.
  5. Debugging with runfiles when run with run_docker.
@nbren12 nbren12 changed the title Add convenience functions for accessing configurations within runfile Conveniences for developing and using runfiles Jan 29, 2020
@nbren12 nbren12 mentioned this issue May 7, 2020
Merged
nbren12 added a commit that referenced this issue May 7, 2020
@nbren12
Capture Fortran-C level outputs to stdout (#65)
8271a4a 
The fortran model produces copious outputs that make it challenging to
debug. This code refactors some functionality from fv3net for capturing
the stdout stream from the fortran model, and remitting it as a debug
level python logging statements.


Here is a sample of the output for `level=logging.DEBUG`:
```
INFO:fv3gfs(0):NOTE from PE     0: Potential error in diag_manager_end: snowwat NOT available, check if output interval > runlength. Netcdf fill_values are written
INFO:fv3gfs(0):NOTE from PE     0: Potential error in diag_manager_end: graupel NOT available, check if output interval > runlength. Netcdf fill_values are written
DEBUG:fv3gfs(0): MPP_DOMAINS_STACK high water mark=       37800
DEBUG:fv3gfs(0):Tabulating mpp_clock statistics across      6 PEs...
DEBUG:fv3gfs(0):                                          tmin          tmax          tavg          tstd  tfrac grain pemin pemax
DEBUG:fv3gfs(0):Total runtime                        11.250342     11.250344     11.250343      0.000000  1.000     0     0     5
DEBUG:fv3gfs(0):Initialization                        1.142568      1.142690      1.142652      0.000042  0.102     0     0     5
DEBUG:fv3gfs(0):FV dy-core                            0.433355      3.979878      1.885681      1.464813  0.168    11     0     5
DEBUG:fv3gfs(0):FV subgrid_z                          0.000504      0.000735      0.000657      0.000086  0.000    11     0     5
DEBUG:fv3gfs(0):FV Diag                               0.011109      0.016041      0.014311      0.002226  0.001    11     0     5
DEBUG:fv3gfs(0):GFS Step Setup                        5.819388      9.333302      7.864650      1.474163  0.699     1     0     5
DEBUG:fv3gfs(0):GFS Radiation                         0.135066      0.339031      0.213256      0.070167  0.019     1     0     5
DEBUG:fv3gfs(0):GFS Physics                           0.035783      0.059737      0.050340      0.010068  0.004     1     0     5
DEBUG:fv3gfs(0):Dynamics get state                    0.002524      0.004121      0.003575      0.000596  0.000     1     0     5
DEBUG:fv3gfs(0):Dynamics update state                 0.023314      3.501858      1.422214      1.453081  0.126     1     0     5
DEBUG:fv3gfs(0):FV3 Dycore                            0.453250      3.993675      1.903505      1.463173  0.169     1     0     5
DEBUG:fv3gfs(0):Main loop                            10.056789     10.056795     10.056791      0.000002  0.894     0     0     5
DEBUG:fv3gfs(0):Termination                           0.046836      0.046848      0.046843      0.000004  0.004     0     0     5
DEBUG:fv3gfs(0): MPP_STACK high water mark=           0
WARNING:fv3gfs(0):WARNING from PE     0: MPP_OPEN: File RESTART/coupler.res opened WRONLY already exists!
```
NOTE, WARNING, and FATAL are logged as INFO, WARNING, and CRITICAL python logging levels. Note that, because errors from fortran are not currently caught by python, this feature will not display any outputs in a subroutine calls the `exit` or `stop` OS routines.

Partial fix to ai2cm/fv3net#246 #32
mcgibbon pushed a commit that referenced this issue Jul 30, 2020
@nbren12
Capture Fortran-C level outputs to stdout (#65)
b0cb3b0 
The fortran model produces copious outputs that make it challenging to
debug. This code refactors some functionality from fv3net for capturing
the stdout stream from the fortran model, and remitting it as a debug
level python logging statements.


Here is a sample of the output for `level=logging.DEBUG`:
```
INFO:fv3gfs(0):NOTE from PE     0: Potential error in diag_manager_end: snowwat NOT available, check if output interval > runlength. Netcdf fill_values are written
INFO:fv3gfs(0):NOTE from PE     0: Potential error in diag_manager_end: graupel NOT available, check if output interval > runlength. Netcdf fill_values are written
DEBUG:fv3gfs(0): MPP_DOMAINS_STACK high water mark=       37800
DEBUG:fv3gfs(0):Tabulating mpp_clock statistics across      6 PEs...
DEBUG:fv3gfs(0):                                          tmin          tmax          tavg          tstd  tfrac grain pemin pemax
DEBUG:fv3gfs(0):Total runtime                        11.250342     11.250344     11.250343      0.000000  1.000     0     0     5
DEBUG:fv3gfs(0):Initialization                        1.142568      1.142690      1.142652      0.000042  0.102     0     0     5
DEBUG:fv3gfs(0):FV dy-core                            0.433355      3.979878      1.885681      1.464813  0.168    11     0     5
DEBUG:fv3gfs(0):FV subgrid_z                          0.000504      0.000735      0.000657      0.000086  0.000    11     0     5
DEBUG:fv3gfs(0):FV Diag                               0.011109      0.016041      0.014311      0.002226  0.001    11     0     5
DEBUG:fv3gfs(0):GFS Step Setup                        5.819388      9.333302      7.864650      1.474163  0.699     1     0     5
DEBUG:fv3gfs(0):GFS Radiation                         0.135066      0.339031      0.213256      0.070167  0.019     1     0     5
DEBUG:fv3gfs(0):GFS Physics                           0.035783      0.059737      0.050340      0.010068  0.004     1     0     5
DEBUG:fv3gfs(0):Dynamics get state                    0.002524      0.004121      0.003575      0.000596  0.000     1     0     5
DEBUG:fv3gfs(0):Dynamics update state                 0.023314      3.501858      1.422214      1.453081  0.126     1     0     5
DEBUG:fv3gfs(0):FV3 Dycore                            0.453250      3.993675      1.903505      1.463173  0.169     1     0     5
DEBUG:fv3gfs(0):Main loop                            10.056789     10.056795     10.056791      0.000002  0.894     0     0     5
DEBUG:fv3gfs(0):Termination                           0.046836      0.046848      0.046843      0.000004  0.004     0     0     5
DEBUG:fv3gfs(0): MPP_STACK high water mark=           0
WARNING:fv3gfs(0):WARNING from PE     0: MPP_OPEN: File RESTART/coupler.res opened WRONLY already exists!
```
NOTE, WARNING, and FATAL are logged as INFO, WARNING, and CRITICAL python logging levels. Note that, because errors from fortran are not currently caught by python, this feature will not display any outputs in a subroutine calls the `exit` or `stop` OS routines.

Partial fix to ai2cm/fv3net#246 #32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nbren12