…ectively.

…es by re-execing NHC after each loop.

Some problems have been reported when NHC is used as a Grid Engine
load sensor, including memory leaks, due to the long-running NHC
process.  All function variables reported as being leaked are local to
their respective functions, but for unknown reasons (BASH bug?),
they're being leaked anyway.

In an attempt to work around these issues, this commit re-execs NHC
after the completion of each iteration.  There are some side effects
to this, including resetting of non-exported environment variables, so
this is experimental for now.

new check: check_nvsmi_healthmon which uses nvidia-smi

… does or does not match a specific node range

…after the last check, such as if pbsnodes hangs and causes nhc to timeout

Added more debug information for if a check matches the node range in nhc.conf

…n works before running tests that need it.

…keeping debugging output added by NREL. Avoids executing mcheck() 3 times on same data.

fixed misleading timeout message if nhc times out after all checks were completed

David Whiteside (GitHub user starboarder2001) noticed (see his PR mej#9
for details) that NHC would sometimes report that it had timed out
while executing a particular check when, in fact, it had already
completed that check and was trying to finish its work and/or clean up
(e.g., marking a node online).

This tweaks his fix to be even more specific about what exactly NHC is
doing as it goes through the final stages of execution.

…bracketing delimiters.

…s the result file.

…o add file/line/function info to PS4 for xtrace output.

…of authorized users.

This fixes mej#15 by using squeue instead of stat to obtain the list of authorized users

…m bbbbbrie to use built-in NHC variable HOSTNAME_S instead of shelling out to the hostname command.

…k depth info.

…ectively.

…es by re-execing NHC after each loop.

Some problems have been reported when NHC is used as a Grid Engine
load sensor, including memory leaks, due to the long-running NHC
process.  All function variables reported as being leaked are local to
their respective functions, but for unknown reasons (BASH bug?),
they're being leaked anyway.

In an attempt to work around these issues, this commit re-execs NHC
after the completion of each iteration.  There are some side effects
to this, including resetting of non-exported environment variables, so
this is experimental for now.

* sge-fixes:
  workaround:  Try avoiding SGE-related memory leaks and other BASH issues by re-execing NHC after each loop.
  Allow SIGUSR1 and SIGUSR2 to toggle bash tracing and debug mode, respectively.

* master:
  Makefile.am:  Minor/cosmetic fix for removing /var/{lib,run}/nhc in uninstall-local rule.
  Allow SLURM nodes in reservation to be marked offline
  Fixed bad links

* master:
  nhc:  Update help output to show correct default value for $TIMEOUT.  Fixes mej#22 from @eshelman.

expansion of wildcards -- especially * -- during parsing of file
content, process listings, etc. (the latest being GH Issue mej#25 from
@barrymoo and @bbenedetto), I finally decided I needed a better, more
fool-proof method of protecting NHC from unintended pathname
expansion.  So I decided to force the issue...literally.

By invoking "set -f" on startup, NHC now disables all pathname (i.e.,
wildcard) expansion throughout its run.  I went through the code and
identified all the locations where globbing files and/or paths was
required, and I've specifically re-enabled the expansion for only
those commands/code blocks that require it, making sure to turn it
back off afterward.

This is going on a dedicated branch for now until I have a chance to
do more thorough testing and vetting of this change since it
potentially impacts how almost every line of code is processed.

…ill Benedetto (@bbenedetto).  This closes mej#25, and similar globbing issues, once and for all!

…asswd` cache. Should fix mej#28 from Trey Dockendorf <[email protected]>.

…nes where they're used.

Some (newer?) versions of BASH don't like creating a process
substitution file on one line and using it on the next while other
(older?) versions seem to handle it just fine.  Keeping everything all
on the same line should work everywhere, so let's do that.  This
should (hopefully?) fix mej#31 which appears on both Fedora and Cygwin64
but which I had previously misunderstood.

* master:
  README.md:  Add blurb and link for SGE/Grid Engine Integration.

ignore specific processes, user names, uids or pids.

Add parameters to check_ps_unauth_users and check_ps_userproc_lineage

* master:
  escaped bash pipe symbols in markdown table
  Make nhc debian aware, fixes mej#38
  Fix typo in check_file_stat

Avoid hangs of NHC when processes are excessively long. Resolves mej#52

Limit length of ps output returned

Set COLUMNS environment variable to 1024 prior to invoking `ps` command to avoid hangs with long command lines.

Only pass check if there is no result, eg:
 *  check_file_test -! -e /etc/node_status/reboot_node

This check will pass if the file is absent

* master:
  check_dmi_data_match():  Use -! for consistency
  README.md:  Add missing documentation
  check_hw_mem():  Fix handling of fudge factor
  Fixed dead link to Torque documentation
  Fix link to source tarball. Should point to releases/download not archive
  Fix incorrect call to `die`

In detached mode, we check to see if we can write to the result file
before actually trying to do so, but `test -w` returns false for
non-existent files, not just unwritable files.  So we need to redo the
logic of that test to account for both cases: (1) file exists and is
writable, or (2) file does not exist but directory does and is
writable.

While I was at it, I also changed where the write is done to account
for a third scenario:  (3) everything looks like it should work but
the write itself fails.

Fixes mej#59.  Thanks to @hwj0303 for spotting this!

* master:
  test:  Add BASH variable tests for clobbering
  check_cmd_output():  Fix index of failed match
  scripts:  Don't clobber $LINENO
  Change to preferred capitialization for Slurm.
  correct example check
  Correct Readme

One of my last major changes before leaving LBNL was to globally
disable pathname expansion (i.e., globbing) by default throughout
NHC.  Apparently when I did that, I missed a spot:  the unit test
driver script!  So only the unit tests specific to `nhc` itself have
been running since that happened (commit 8fa6657).

While this problem has existed for over a year-and-a-half in wallclock
time, commit-wise it hasn't been very long.  (About 10-ish, not
counting merging work from others.)  That's my story, and I'm sticking
to it.

Fix the process substitution sanity check so that we're not skipping
46 unit tests for no reason whatsoever.

check_file_test added negative option

While taking another look at @SMark-Black's mej#50 and mej#53, I realized
that the code in question regarding `$MAX_SYS_UID` is doing exactly
what it is supposed to do, given the intended meaning of the variable
based on its name.  What was *actually* wrong was that the
`nhc_common_get_max_sys_uid()` function has been reading the wrong
variable!

So `nhc_common_get_max_sys_uid()` will now look for `$SYS_UID_MAX` in
`/etc/login.defs` like it should have been doing all along, and using
the value of `UID_MIN - 1` as a fallback if necessary.

NOTE:  This means that the default auto-detected value of
`$MAX_SYS_UID` will likely be something ending in `99` (like `499` or
`999`) rather than `00` because it was always intended to be (and the
code has always treated it as) the *top* of the exempt UID range, NOT
the bottom of the non-exempt UID range!  If you have any configs or
scripts that rely on different assumptions, please make sure to make
any necessary updates.

Closes mej#50.

Based on a couple changes suggested by @SMark-Black in his PR mej#53, add
another command to look for to auto-detect LSF, and add support for
the LSF `res` daemon to the `check_ps_userproc_lineage()` check.  Also
moved the setting of `$RM_DAEMON_MATCH` to inside the check -- that's
the only thing in that whole entire file that actually requires a
resource manager!

Disable the logfile when in eval (`-e`) mode so that NHC doesn't try
to output to both the log and `stdout` and wind up saying the same
thing twice!

Add a Table of Contents to the README.md documentation file
automatically generated by the `gh-md-toc` script from @ekalinin.

Many thanks to @basvandervlies for both suggesting this much-needed
addition and helping find an editor-agnostic way to generate it
automatically going forward!

To update:
```bash
$ git clone https://github.com/ekalinin/github-markdown-toc.git
$ github-markdown-toc/gh-md-toc --insert README.md
```

Closes mej#67.  Feedback is welcome on (1) whether or not to
submodule-ize this, and (2) whether or not any changes are needed to
tweak the output for NHC...it looks like it might need some manual
tweaking at the moment.

Made a couple tweaks to the `gh-md-toc` script (NHC-specific) to put
the Table of Contents more in line with what the intent of the
formatting was, not necessarily the exact indentation level.  I
intentionally skip heading levels to achieve the correct style, but
that confuses the ToC generator.  Some special casing in the `awk`
script has remedied that.

I haven't taken the time to make the changes in a way that would be
generic enough to consider upstreaming.  Maybe in the future!

Conflicts:
	nhc