Observing transient timeouts during KayVee distributed-store healthcheck

[allengeorge]

The KayVee distributed-store health-check has a command-commit timeout of 500ms. This should be more than enough to get consensus and commit log entries to disk for a 3-machine cluster. I notice, however, transient timeouts in running a health-check loop (1 every second). There are sometimes spikes when the check is being run against the leader and the leader has to deal with a faulty machine (see #11).

[allengeorge]

This may be due to transient load spikes on my local machine. I notice that my dev machine experiences multi-second pauses (presumably paging something to/from disk) during day-to-day work.

[allengeorge]

I can't reproduce this with the set of long-term low-load soak tests I've run on GCE. I'm going to chalk it up to running n nodes on my iMac with lowish amounts of RAM.