How to RUN a scala file(OnDemandDruidExhaustJob) in Sunbird Obsrv core-dataproduct without using testcase ? #109

sumanvpacewisdom · 2023-08-21T07:41:38Z

sumanvpacewisdom
Aug 21, 2023

I have installated core-dataproduct from Sunbird-Obsrv Building block and able to trigger OnDemandDruidExhaustJob via testfile called TestOnDemandDruidExhaustJob.scala , But how do i trigger OnDemandDruidExhaustJob without testcases.

I have the following system/software config

OS :- Ventura [Mac with M1 silicon]
Maven :- Apache Maven 3.9.4
Java :- openjdk 11.0.19 [Same version is used in IDE]
Scala :- Scala 2.12.10 [Same version is used in IDE]
Spark :- Spark 3.4.1

Steps that I followed

Cloned sunbird-core-dataproducts and checked out to the branch 5.1.1

git clone https://github.com/Sunbird-Obsrv/sunbird-core-dataproducts.git
git checkout release-5.1.1

Built sunbird-core-dataproducts via following command [BUILD SUCCESS ✅]

mvn clean install -DskipTests

I have set a local spark and trying to submit the OnDemandDruidExhaustJob inside the spark by giving below command with required jar files to run the spark-job

./bin/spark-submit --class org.ekstep.analytics.exhaust.OnDemandDruidExhaustJob \
 --master 'local[*]' \
 /Users/user/.m2/repository/org/sunbird/batch-models/2.0/batch-models-2.0.jar \
 /Users/user/Documents/shikshalokam/tools/job-config.json

Error after running the above command

Exception in thread "main" java.lang.NoClassDefFoundError: org/ekstep/analytics/framework/FrameworkContext
	at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
	at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3166)
	at java.base/java.lang.Class.getMethodsRecursive(Class.java:3307)
	at java.base/java.lang.Class.getMethod0(Class.java:3293)
	at java.base/java.lang.Class.getMethod(Class.java:2106)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:42)
	at [org.apache.spark.deploy.SparkSubmit.org](http://org.apache.spark.deploy.sparksubmit.org/)$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.ekstep.analytics.framework.FrameworkContext
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 13 more
23/08/22 16:56:56 INFO ShutdownHookManager: Shutdown hook called
23/08/22 16:56:56 INFO ShutdownHookManager: Deleting directory /private/var/folders/f7/kt5z7zpj61v5b757h_wxcyh80000gn/T/spark-1e545bac-ee5e-46f9-b893-4ae297437a0d

@manjudr - Can you please help here.

Cc - @vijiurs @aks30 @kiranharidas187 @Vivek-M-08 @rakeshSgr

Answered by manjudr

Aug 29, 2023

What is the job are you trying to execute?
For example, If you are running the "wfs" job, all the job configurations should be defined in the model-config.sh and also define the model name in the case class of the run-job.sh

Command to run the job - nohup ./run-job.sh wfs $

View full answer

manjudr · 2023-08-25T05:12:45Z

manjudr
Aug 25, 2023
Maintainer

@sumanvpacewisdom - did you include the analytics-core jar dependency in your run job script?

2 replies

sumanvpacewisdom Aug 25, 2023
Author

After Including the analytics-core, analytics-job-driver, batch-models-2.0., original-batch-models-2.0, original-analytics-framework-2.0, analytics-framework-2.0 jars dependency with using below command

./bin/spark-submit --class org.ekstep.analytics.exhaust.OnDemandDruidExhaustJob \
  --master 'local[*]' \
/Users/sumanv/Documents/ShikshaLokam/Projects/ml-exhaust/sunbird-core-dataproducts/batch-models/target/batch-models-2.0.jar \
/Users/sumanv/Documents/ShikshaLokam/Projects/ml-exhaust/sunbird-core-dataproducts/batch-models/target/original-batch-models-2.0.jar \
/Users/sumanv/Documents/ShikshaLokam/Projects/ml-exhaust/sunbird-analytics-core/analytics-job-driver/target/analytics-framework-2.0.jar \
/Users/sumanv/Documents/ShikshaLokam/Projects/ml-exhaust/sunbird-analytics-core/analytics-job-driver/target/original-analytics-framework-2.0.jar \
/Users/sumanv/Documents/ShikshaLokam/Projects/ml-exhaust/sunbird-analytics-core/analytics-core/target/analytics-core-2.0.jar \
/Users/sumanv/Documents/ShikshaLokam/Projects/ml-exhaust/sunbird-core-dataproducts/job-config.json

Error for above command after including the dependencies

23/08/25 11:41:20 WARN Utils: Your hostname, PWA166.local resolves to a loopback address: 127.94.0.2; using 192.168.0.101 instead (on interface en0)
23/08/25 11:41:20 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Exception in thread "main" java.lang.NoClassDefFoundError: org/ekstep/analytics/framework/FrameworkContext
at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3166)
at java.base/java.lang.Class.getMethodsRecursive(Class.java:3307)
at java.base/java.lang.Class.getMethod0(Class.java:3293)
at java.base/java.lang.Class.getMethod(Class.java:2106)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:42)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.ekstep.analytics.framework.FrameworkContext
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 13 more
23/08/25 11:41:25 INFO ShutdownHookManager: Shutdown hook called
23/08/25 11:41:25 INFO ShutdownHookManager: Deleting directory /private/var/folders/0w/4_s7903s36d4jbhnxl0yvvzw0000gn/T/spark-54649647-8f8c-4c84-8480-9b07e40dd2b7

Error After runing the above comand for second time

23/08/25 11:52:30 WARN Utils: Your hostname, PWA166.local resolves to a loopback address: 127.94.0.2; using 192.168.0.101 instead (on interface en0)
23/08/25 11:52:30 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Exception in thread "main" java.lang.NoClassDefFoundError: org/ekstep/analytics/framework/JobConfig
at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3166)
at java.base/java.lang.Class.getMethodsRecursive(Class.java:3307)
at java.base/java.lang.Class.getMethod0(Class.java:3293)
at java.base/java.lang.Class.getMethod(Class.java:2106)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:42)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.ekstep.analytics.framework.JobConfig
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 13 more
23/08/25 11:52:30 INFO ShutdownHookManager: Shutdown hook called
23/08/25 11:52:30 INFO ShutdownHookManager: Deleting directory /private/var/folders/0w/4_s7903s36d4jbhnxl0yvvzw0000gn/T/spark-8f7c7634-bb9b-442c-b149-bb53dbe72e6b

@manjudr After Including the jars dependencies, i am not unable to procced further, please help me with further steps to resolve.

manjudr Aug 25, 2023
Maintainer

Why so many analytics core jar files? All we need to mention analytics-framework-2.0.ja, scruid_2.12-2.5.0.jar batch-models-2.0.jar

Please refer to this shell script to run the data products and please ensure updating all the variables in the script before running the job - https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/blob/master/ansible/roles/data-products-deploy/templates/run-job.j2.

sumanvpacewisdom · 2023-08-28T12:13:55Z

sumanvpacewisdom
Aug 28, 2023
Author

According to the shell script , I have updated the variables. Here is the used to run in local Run Script

Error For above script

Starting the job - 
Job modelName - 
23/08/28 15:42:49 WARN Utils: Your hostname, PWA166.local resolves to a loopback address: 127.94.0.2; using 192.168.0.101 instead (on interface en0)
23/08/28 15:42:49 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
23/08/28 15:42:50 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
inside Job Executor main method
Exception in thread "main" org.ekstep.analytics.framework.exception.JobNotFoundException: Unknown job type found
	at org.ekstep.analytics.job.JobFactory$.reflectModule(JobFactory.scala:52)
	at org.ekstep.analytics.job.JobFactory$.getJob(JobFactory.scala:40)
	at org.ekstep.analytics.job.JobExecutor$.main(JobExecutor.scala:15)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at optional.Application.callWithOptions(application.scala:251)
	at optional.Application.callWithOptions$(application.scala:223)
	at org.ekstep.analytics.job.JobExecutor$.callWithOptions(JobExecutor.scala:8)
	at optional.Application.main(application.scala:258)
	at optional.Application.main$(application.scala:255)
	at org.ekstep.analytics.job.JobExecutor$.main(JobExecutor.scala:8)
	at org.ekstep.analytics.job.JobExecutor.main(JobExecutor.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
23/08/28 15:42:51 INFO ShutdownHookManager: Shutdown hook called
23/08/28 15:42:51 INFO ShutdownHookManager: Deleting directory /private/var/folders/0w/4_s7903s36d4jbhnxl0yvvzw0000gn/T/spark-fa9d6e20-204b-4d53-a7b9-1098d7e4073b
Job execution completed -

In Job Exceuter in main function there we need to get the job using jobFactory class and in this class the above exception is thrown. I was able to find where it is casuing error , I tried to resolve it by using the model-config template but i got the same above error. I was unable to process further .

@manjudr Can you help me through steps, which i feel would be great, so it would easy to understanding and work on it .

1 reply

manjudr Aug 29, 2023
Maintainer

What is the job are you trying to execute?
For example, If you are running the "wfs" job, all the job configurations should be defined in the model-config.sh and also define the model name in the case class of the run-job.sh

Command to run the job - nohup ./run-job.sh wfs $

Answer selected by manjudr

sumanvpacewisdom · 2023-09-01T08:16:25Z

sumanvpacewisdom
Sep 1, 2023
Author

This Issue is resolved .

@manjudr Thank you for solving my issue this helped me to run the job needed.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sunbird Obsrv

How to RUN a scala file(OnDemandDruidExhaustJob) in Sunbird Obsrv core-dataproduct without using testcase ? #109

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Sunbird Obsrv

How to RUN a scala file(OnDemandDruidExhaustJob) in Sunbird Obsrv core-dataproduct without using testcase ? #109

sumanvpacewisdom Aug 21, 2023

I have the following system/software config

Steps that I followed

Error after running the above command

Replies: 3 comments · 3 replies

manjudr Aug 25, 2023 Maintainer

sumanvpacewisdom Aug 25, 2023 Author

After Including the analytics-core, analytics-job-driver, batch-models-2.0., original-batch-models-2.0, original-analytics-framework-2.0, analytics-framework-2.0 jars dependency with using below command

Error for above command after including the dependencies

Error After runing the above comand for second time

manjudr Aug 25, 2023 Maintainer

sumanvpacewisdom Aug 28, 2023 Author

According to the shell script , I have updated the variables. Here is the used to run in local Run Script

Error For above script

manjudr Aug 29, 2023 Maintainer

sumanvpacewisdom Sep 1, 2023 Author

sumanvpacewisdom
Aug 21, 2023

Replies: 3 comments 3 replies

manjudr
Aug 25, 2023
Maintainer

sumanvpacewisdom Aug 25, 2023
Author

manjudr Aug 25, 2023
Maintainer

sumanvpacewisdom
Aug 28, 2023
Author

manjudr Aug 29, 2023
Maintainer

sumanvpacewisdom
Sep 1, 2023
Author