Add verbosity to smart-dispatch #175
Conversation
It is difficult to debug resuming while important process are taking place in the pbs script automatically built by SmartDispatch. We add verbose to smart-dispatch script and add debugging prints in epilog.
|
@mgermain Small PR I took out of Slurm Backend PR |
|
|
||
| logger = logging.getLogger() | ||
|
|
||
|
|
There was a problem hiding this comment.
remove the logging.root.setLevel(logging.INFO) below and make sure the default throughout the system is warning.
There was a problem hiding this comment.
Even in the rest of the smartdispatch to stay consistent?
The user can now specify which level of logging it wants with -vv option. The default one is WARNING. If is it important for the user to have INFO present in stderr, it can use -v or -vv.
scripts/smart-dispatch
Outdated
| @@ -29,19 +33,45 @@ TIMEOUT_EXIT_CODE = 124 | |||
| AUTORESUME_TRIGGER_AFTER = '$(($PBS_WALLTIME - 60))' # By default, 60s before the maximum walltime. | |||
| AUTORESUME_WORKER_CALL_PREFIX = 'timeout -s TERM {trigger_after} '.format(trigger_after=AUTORESUME_TRIGGER_AFTER) | |||
| AUTORESUME_WORKER_CALL_SUFFIX = ' WORKER_PIDS+=" $!"' | |||
| AUTORESUME_PROLOG = 'WORKER_PIDS=""' | |||
| AUTORESUME_PROLOG = """ | |||
There was a problem hiding this comment.
Add the \ after the """ so there is no extra blank line later on.
AUTORESUME_PROLOG = """\
scripts/smart-dispatch
Outdated
| epilog = [AUTORESUME_EPILOG.format(launcher=LAUNCHER if args.launcher is None else args.launcher, path_job=path_job)] | ||
| prolog = [ | ||
| AUTORESUME_PROLOG.format(verbose=str(args.verbose >= 2).lower())] | ||
| epilog = [ |
There was a problem hiding this comment.
This project follows the entirety of PEP8 but the 80 columns.
|
After the tiny modifs mentioned this is ready to be merged. |
I've been told the admins decided long ago not to follow rule E501. Even though I disagree with this decision, I won't argue. :P
VERBOSE in the PBS file was simply set to "true". Because of this we could only
pass args.verbose=0 or args.verbose=2 to sd-launch-pbs. VERBOSE is now set to
args.verbose and the command-line option {"", -v, -vv} is rebuilt in the PBS
script based on the value of args.parse.
|
@mgermain I bit the bullet and reverted the E501 fixes. 😜 I also changed the debugging logic in the epilog to support passing WARNING/INFO/DEBUG levels to sd-launch-pbs. Previously, it could only pass WARNING (no -v) or DEBUG (-vv). |
It is difficult to debug resuming while important process are taking place in the pbs script automatically built by SmartDispatch.
Adding: