Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not able to find the cause of failure of HDInsight Script action. #12

Open
sumanentc opened this issue May 18, 2018 · 0 comments
Open

Comments

@sumanentc
Copy link

Hi ,
i used custom script in ARM Template while creating HDInsight Spark Cluster. The template was getting executed successfully , but the script was not getting executed. I used some environment variables in my script which was not getting resolved. I spent a lot of time to figure it out the issue.
"scriptActions": [{
"name": "createLivyFolder",
"uri": "[parameters('scriptActionUri')]"
}]
The below challenges i faced

  1. Not able to get the path of the folder from where the script was executed in HDInsight Cluster.
  2. Not able to get the path of error log or Stack-trace of script action.

After spending a lot of time i got the log from the Azure blob storage and was able to get the cause of the problem.
Can you please add to the doc how to debug the script action in case of failure or get the stack trace in case of success/failure.

Please let us know if we want to use some environment variables in script action like $SPARK_HOME,
How to do that. The below statement was failing in script actions.
sudo cp postgresql-9.4.1210.jre7.jar $SPARK_HOME/jars/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant