StackOverflowError when returning a successful result

[joe-niland]

Hey there,

Thanks for an amazing plugin.

I think I have a similar issue to #106 and possibly #101 except it's occurring with the runner interface.

I'm using the Salt API to orchestrate some long running tasks and, for some reason, this issue does not occur on every run.

What I'm seeing is the plugin logs that it receives the response:

saltstack-plugin/src/main/java/com/waytta/SaltAPIStep.java

Line 331 in e863eae

LOGGER.log(Level.FINE, "Received response: " + returnArray);

and then shortly after an exception is thrown in the main Jenkins logs (I think it's on this line:

saltstack-plugin/src/main/java/com/waytta/SaltAPIStep.java

Line 349 in e863eae

getContext().onSuccess(returnArray.toString());

Oct 05, 2017 8:48:52 AM SEVERE hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException
A thread (saltAPI/20467) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
java.lang.StackOverflowError
	at org.apache.commons.beanutils.PropertyUtilsBean.getPropertyDescriptors(PropertyUtilsBean.java:1084)
	at org.apache.commons.beanutils.PropertyUtilsBean.getPropertyDescriptor(PropertyUtilsBean.java:912)
	at org.apache.commons.beanutils.PropertyUtilsBean.getSimpleProperty(PropertyUtilsBean.java:1319)
	at org.apache.commons.beanutils.PropertyUtilsBean.getNestedProperty(PropertyUtilsBean.java:770)
	at org.apache.commons.beanutils.PropertyUtilsBean.getProperty(PropertyUtilsBean.java:846)
	at org.apache.commons.beanutils.PropertyUtils.getProperty(PropertyUtils.java:426)
	at net.sf.json.JSONObject.defaultBeanProcessing(JSONObject.java:692)
	at net.sf.json.JSONObject._fromBean(JSONObject.java:642)
	at net.sf.json.JSONObject.fromObject(JSONObject.java:169)
	at net.sf.json.JSONSerializer.toJSON(JSONSerializer.java:108)
	at net.sf.json.AbstractJSON._processValue(AbstractJSON.java:239)
	at net.sf.json.JSONArray._processValue(JSONArray.java:2492)
	at net.sf.json.JSONArray.processValue(JSONArray.java:2517)
	at net.sf.json.JSONArray.addValue(JSONArray.java:2504)
	at net.sf.json.JSONArray.element(JSONArray.java:1719)
	at net.sf.json.JSONArray.fromObject(JSONArray.java:159)
	at net.sf.json.JSONSerializer.toJSON(JSONSerializer.java:113)
	at net.sf.json.AbstractJSON._processValue(AbstractJSON.java:239)
        ...
        etc
        ...

Jenkins does not see the build as failed and if I leave it, it will run seemingly indefinitely.

It is returning a lot of JSON (from a state that creates an EC2 instance) but nowhere near as much as in your 'bigReturn.json' test file so I can't yet understand why it would have an issue with this return. I even copied my json into a json file and ran mvn test with no errors.

I can send you the JSON response from Salt-API separately if you like.

Any ideas on how I can better troubleshoot this?

Thanks!

[mchugh19]

Wow, sounds tricky. So the jenkins console shows the return from salt, and then afterwards throws the exception? Are you using freestyle jobs, or just pipeline?

As for next debugging steps, my only thought is to add a bunch of debug print statements after the final output print to see what is exception'ing.

In an effort to attempt to replicate, could you send me what you are using for a job, the log output, and the json that is returned? You can email my username at hotmail.com

[joe-niland]

Hi @mchugh19

Thanks for your message.

The Jenkins console shows only the call to the saltstack plugin and then hangs at that point.
The exception can be seen in the "All Jenkins logs" recorder.

For this one we're just using the pipeline (scripted).

Do you mean recompile your plugin with the extra debug statements? I can definitely try that.

Something I forgot to mention - this issue is only evident when running a few of these builds in parallel. When running one build at a time, I haven't seen this issue occur. I wonder if it's nothing to do with the plugin but maybe with the pipeline system instead.

I will send the info you mentioned ASAP. I'll try to gather more clues first!

Thanks again

[joe-niland]

Hi @mchugh19

I was just working on creating a reproducible example and now cannot reproduce it!
Today I also upgraded Jenkins to 2.84 so not sure if that's related. I think this issue can be closed for now.

One question, though: I was trying to recompile and add extra debug statements, however when I uploaded my compiled hpi file, Jenkins would not recognise the 'salt' keyword:

java.lang.NoSuchMethodError: No such DSL method 'salt' found among steps

Is there anything else to do other than mvn package then uploading the hpi file in the Plugin management area?

I may have found a bug in com.waytta.Builds.getBlockingBuildJid so I'd like to be able to test my own version of the plugin in order to resolve this.

Thanks!

[mchugh19]

Thanks for the investigations!

I have a new laptop, so over the last few days I've been setting up the development environment. In doing so, I've been changing jenkins versions. When using the latest jenkins, I am also unable to replicate. What version were you on before the upgrade?

As for the DSL error, that's one I've not seen before. You should just be able to use the hpi from mvn package. The salt pipeline step should be available from the public class SaltAPIStep extends Step and descriptor in SaltAPIStep.java.

If you have any more thoughts about the getBlockBuildJid issue, I'd be happy to help.

Thanks again

[joe-niland]

np, we were on 2.82 and then upgraded to 2.83 today.

Yeah I am not sure what's going wrong with the hpi file. Perhaps there's some Jenkins config or other issue that's messing it up. I will keep trying.

Feel free to close this for now, but I will definitely let you know if I can reproduce this issue.

I've put details about the issue with getBlockingBuildJid in #109.

[mchugh19]

Based on comments in #101, I think we may have found a cause for this issue. The http call to get the response from the salt-api only waits 5 seconds, and if it is a large response, this might not be enough. I've just upped it to 30 in this release. Could test and see if this helps?

[joe-niland]

Thanks for following up. I'll test this again ASAP and let you know how it goes.

[joe-niland]

Hey @mchugh19
I just re-tested using my standard test case for this issue.
All calls to Salt-API returned within the expected time, so this is good.

Are you able to elaborate a little on why increasing the conn timeout to 30s makes a difference?

[mchugh19]

My thought was simply that if the return from the salt-api was very large, it might not have completed in a short 5 second window.

[joe-niland]

OK, great, thanks.
So far so good with this update.

Also, when will 3.1.3 go out to the Jenkins plugin repo?

[mchugh19]

Just published the release. It should hit the mirrors within the next 24 hours.

I'll close this out for now, but if you have any issues, please re-open. Thanks!

[richardsimko]

Hey! Did you ever publish 3.1.3? The latest version reported is still 3.1.2

[joe-niland]

@richardsimko 3.1.3 is showing up for us in the Jenkins plugin manager

[richardsimko]

Nevermind, we already had 3.1.3, that's why it wouldn't show up! However the latest version on the Wiki says 3.1.2 but maybe that's manually updated?