Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

scp-receive-command might fails with AIOOB #59

Open
frozenspider opened this issue Jul 9, 2019 · 1 comment
Open

scp-receive-command might fails with AIOOB #59

frozenspider opened this issue Jul 9, 2019 · 1 comment

Comments

@frozenspider
Copy link

(Created from jepsen-io/jepsen#398)
Hey,
We're using Jepsen 0.1.14 which in turn uses clj-ssh-0.5.14. Here's a stack trace I ran into:

WARN [2019-07-09 11:53:31,619] main - jepsen.core Test crashed!
java.lang.ArrayIndexOutOfBoundsException: -2
	at clj_ssh.ssh$scp_receive_command$fn__2679.invoke(ssh.clj:874) ~[jepsen-0.1.14.jar:na]
	at clj_ssh.ssh$scp_receive_command.invokeStatic(ssh.clj:869) ~[jepsen-0.1.14.jar:na]
	at clj_ssh.ssh$scp_receive_command.invoke(ssh.clj:864) ~[jepsen-0.1.14.jar:na]
	at clj_ssh.ssh$scp_sink.invokeStatic(ssh.clj:980) ~[jepsen-0.1.14.jar:na]
	at clj_ssh.ssh$scp_sink.invoke(ssh.clj:977) ~[jepsen-0.1.14.jar:na]
	at clj_ssh.ssh$scp_from.invokeStatic(ssh.clj:1094) ~[jepsen-0.1.14.jar:na]
	at clj_ssh.ssh$scp_from.doInvoke(ssh.clj:1044) ~[jepsen-0.1.14.jar:na]
	at clojure.lang.RestFn.invoke(RestFn.java:445) ~[clojure-1.10.0.jar:na]
	at clojure.lang.AFn.applyToHelper(AFn.java:160) ~[clojure-1.10.0.jar:na]
	at clojure.lang.RestFn.applyTo(RestFn.java:132) ~[clojure-1.10.0.jar:na]
	at clojure.core$apply.invokeStatic(core.clj:667) ~[clojure-1.10.0.jar:na]
	at clojure.core$apply.invoke(core.clj:660) ~[clojure-1.10.0.jar:na]
	at jepsen.control$download$fn__2881.invoke(control.clj:227) ~[jepsen-0.1.14.jar:na]
	at jepsen.control$download.invokeStatic(control.clj:227) ~[jepsen-0.1.14.jar:na]
	at jepsen.control$download.doInvoke(control.clj:223) ~[jepsen-0.1.14.jar:na]
	at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.10.0.jar:na]
	at jepsen.core$snarf_logs_BANG_$fn__5395$fn__5414.invoke(core.clj:133) ~[jepsen-0.1.14.jar:na]
	at jepsen.core$snarf_logs_BANG_$fn__5395.invoke(core.clj:132) ~[jepsen-0.1.14.jar:na]
	at jepsen.control$on_nodes$fn__2918.invoke(control.clj:391) ~[jepsen-0.1.14.jar:na]
	at clojure.lang.AFn.applyToHelper(AFn.java:154) ~[clojure-1.10.0.jar:na]
	at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.10.0.jar:na]
	at clojure.core$apply.invokeStatic(core.clj:665) ~[clojure-1.10.0.jar:na]
	at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973) ~[clojure-1.10.0.jar:na]
	at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973) ~[clojure-1.10.0.jar:na]
	at clojure.lang.RestFn.applyTo(RestFn.java:142) ~[clojure-1.10.0.jar:na]
	at clojure.core$apply.invokeStatic(core.clj:669) ~[clojure-1.10.0.jar:na]
	at clojure.core$bound_fn_STAR_$fn__5734.doInvoke(core.clj:2003) ~[clojure-1.10.0.jar:na]
	at clojure.lang.RestFn.invoke(RestFn.java:408) ~[clojure-1.10.0.jar:na]
	at dom_top.core$real_pmap_helper$build_thread__214$fn__215.invoke(core.clj:146) ~[jepsen-0.1.14.jar:na]
	at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.10.0.jar:na]
	at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.10.0.jar:na]
	at clojure.core$apply.invokeStatic(core.clj:665) ~[clojure-1.10.0.jar:na]
	at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973) ~[clojure-1.10.0.jar:na]
	at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973) ~[clojure-1.10.0.jar:na]
	at clojure.lang.RestFn.invoke(RestFn.java:425) ~[clojure-1.10.0.jar:na]
	at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.10.0.jar:na]
	at clojure.lang.RestFn.applyTo(RestFn.java:132) ~[clojure-1.10.0.jar:na]
	at clojure.core$apply.invokeStatic(core.clj:669) ~[clojure-1.10.0.jar:na]
	at clojure.core$bound_fn_STAR_$fn__5734.doInvoke(core.clj:2003) ~[clojure-1.10.0.jar:na]
	at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.10.0.jar:na]
	at clojure.lang.AFn.run(AFn.java:22) ~[clojure-1.10.0.jar:na]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171]
ERROR [2019-07-09 11:53:31,629] main - jepsen.cli Oh jeez, I'm sorry, Jepsen broke. Here's why:
java.lang.ArrayIndexOutOfBoundsException: -2
	at clj_ssh.ssh$scp_receive_command$fn__2679.invoke(ssh.clj:874) ~[jepsen-0.1.14.jar:na]
	at clj_ssh.ssh$scp_receive_command.invokeStatic(ssh.clj:869) ~[jepsen-0.1.14.jar:na]
	at clj_ssh.ssh$scp_receive_command.invoke(ssh.clj:864) ~[jepsen-0.1.14.jar:na]
	at clj_ssh.ssh$scp_sink.invokeStatic(ssh.clj:980) ~[jepsen-0.1.14.jar:na]
	at clj_ssh.ssh$scp_sink.invoke(ssh.clj:977) ~[jepsen-0.1.14.jar:na]
	at clj_ssh.ssh$scp_from.invokeStatic(ssh.clj:1094) ~[jepsen-0.1.14.jar:na]
	at clj_ssh.ssh$scp_from.doInvoke(ssh.clj:1044) ~[jepsen-0.1.14.jar:na]
	at clojure.lang.RestFn.invoke(RestFn.java:445) ~[clojure-1.10.0.jar:na]
	at clojure.lang.AFn.applyToHelper(AFn.java:160) ~[clojure-1.10.0.jar:na]
	at clojure.lang.RestFn.applyTo(RestFn.java:132) ~[clojure-1.10.0.jar:na]
	at clojure.core$apply.invokeStatic(core.clj:667) ~[clojure-1.10.0.jar:na]
	at clojure.core$apply.invoke(core.clj:660) ~[clojure-1.10.0.jar:na]
	at jepsen.control$download$fn__2881.invoke(control.clj:227) ~[jepsen-0.1.14.jar:na]
	at jepsen.control$download.invokeStatic(control.clj:227) ~[jepsen-0.1.14.jar:na]
	at jepsen.control$download.doInvoke(control.clj:223) ~[jepsen-0.1.14.jar:na]
	at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.10.0.jar:na]
	at jepsen.core$snarf_logs_BANG_$fn__5395$fn__5414.invoke(core.clj:133) ~[jepsen-0.1.14.jar:na]
	at jepsen.core$snarf_logs_BANG_$fn__5395.invoke(core.clj:132) ~[jepsen-0.1.14.jar:na]
	at jepsen.control$on_nodes$fn__2918.invoke(control.clj:391) ~[jepsen-0.1.14.jar:na]
	at clojure.lang.AFn.applyToHelper(AFn.java:154) ~[clojure-1.10.0.jar:na]
	at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.10.0.jar:na]
	at clojure.core$apply.invokeStatic(core.clj:665) ~[clojure-1.10.0.jar:na]
	at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973) ~[clojure-1.10.0.jar:na]
	at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973) ~[clojure-1.10.0.jar:na]
	at clojure.lang.RestFn.applyTo(RestFn.java:142) ~[clojure-1.10.0.jar:na]
	at clojure.core$apply.invokeStatic(core.clj:669) ~[clojure-1.10.0.jar:na]
	at clojure.core$bound_fn_STAR_$fn__5734.doInvoke(core.clj:2003) ~[clojure-1.10.0.jar:na]
	at clojure.lang.RestFn.invoke(RestFn.java:408) ~[clojure-1.10.0.jar:na]
	at dom_top.core$real_pmap_helper$build_thread__214$fn__215.invoke(core.clj:146) ~[jepsen-0.1.14.jar:na]
	at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.10.0.jar:na]
	at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.10.0.jar:na]
	at clojure.core$apply.invokeStatic(core.clj:665) ~[clojure-1.10.0.jar:na]
	at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973) ~[clojure-1.10.0.jar:na]
	at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973) ~[clojure-1.10.0.jar:na]
	at clojure.lang.RestFn.invoke(RestFn.java:425) ~[clojure-1.10.0.jar:na]
	at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.10.0.jar:na]
	at clojure.lang.RestFn.applyTo(RestFn.java:132) ~[clojure-1.10.0.jar:na]
	at clojure.core$apply.invokeStatic(core.clj:669) ~[clojure-1.10.0.jar:na]
	at clojure.core$bound_fn_STAR_$fn__5734.doInvoke(core.clj:2003) ~[clojure-1.10.0.jar:na]
	at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.10.0.jar:na]
	at clojure.lang.AFn.run(AFn.java:22) ~[clojure-1.10.0.jar:na]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171]
@frozenspider frozenspider mentioned this issue Jul 9, 2019
Open
ns-codereview pushed a commit to couchbaselabs/jepsen.couchbase that referenced this issue Jul 18, 2019
@rdemellow
MB-35131 Workaround clj-ssh issue #59
74ed912 
clj-commons/clj-ssh#59 appears to be causing
issues where our tests crash. Inject a retry loop into Jepsen's
download function to detect and recover from instances of this error.

Change-Id: I6f50a08969c2cbdf139c70f7485770858623b8e9
Reviewed-on: http://review.couchbase.org/112147
Tested-by: Build Bot <[email protected]>
Reviewed-by: Richard de Mellow <[email protected]>
@jll63
Copy link

jll63 commented Oct 2, 2019

I am encountering this issue as well, also while running Jepsen. Is work planned on this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@frozenspider @jll63