MB-35131 Workaround clj-ssh issue #59

clj-commons/clj-ssh#59 appears to be causing issues where our tests crash. Inject a retry loop into Jepsen's download function to detect and recover from instances of this error. Change-Id: I6f50a08969c2cbdf139c70f7485770858623b8e9 Reviewed-on: http://review.couchbase.org/112147 Tested-by: Build Bot <[email protected]> Reviewed-by: Richard de Mellow <[email protected]>
couchbaselabs · Jul 18, 2019 · 74ed912 · 74ed912
1 parent b2c5313
commit 74ed912
Showing 1 changed file with 19 additions and 3 deletions.
diff --git a/src/couchbase/core.clj b/src/couchbase/core.clj
@@ -4,6 +4,7 @@
             [couchbase
              [util     :as util]
              [workload :as workload]]
+            [dom-top.core :refer [with-retry]]
             [jepsen
              [cli     :as cli]
              [control :as c]
@@ -263,8 +264,23 @@
      (fn [output-path]
        (assoc-in (vec (preamble output-path)) [1 5 :xs] '(1800 800)))))
 
+  ;; We've encountered crashed during log collection that appear to be caused
+  ;; by https://github.com/hugoduncan/clj-ssh/issues/59. Until it is fixed,
+  ;; inject a retry loop for that issue, and just move on if we keep failing.
+  (alter-var-root
+   (var jepsen.control/download)
+   (fn [download]
+     (fn [& args]
+       (with-retry [attempts 5]
+         (apply download args)
+         (catch ArrayIndexOutOfBoundsException _
+           (warn "Encountered clj-ssh issue #59 during log download")
+           (if (pos? attempts)
+             (retry (dec attempts))
+             (error "Log download failed due to clj-ssh issue #59")))))))
+
   ;; Now parse args and run the test
-  (let [test        (cli/single-test-cmd {:test-fn  cbtest
-                                          :opt-spec extra-cli-options})
-        serve       (cli/serve-cmd)]
+  (let [test (cli/single-test-cmd {:test-fn  cbtest
+                                   :opt-spec extra-cli-options})
+        serve (cli/serve-cmd)]
     (cli/run! (merge test serve) args)))