Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Policy Validation #1861

Open
vardancse opened this issue May 9, 2017 · 3 comments
Open

Policy Validation #1861

vardancse opened this issue May 9, 2017 · 3 comments

Comments

@vardancse
Copy link

Created a policy which pulls message from Kafka topic and extract one field of out of the message and push it to parquet file. Policy was created fine but it couldn't succeed. Below is the error and attach is the policy. Can someone help what's misconfigured? I'm running on standalone mode.
kakfa_cabinfo.txt

09 May 2017 07:48:51.388 INFO c.s.s.d.s.StreamingContextService Listener added to kakfa_cabinfo with id: aa4d828e-5d2a-4ed0-9dc7-784b0bdef43d
09 May 2017 07:48:51.388 INFO c.s.s.d.s.StreamingContextService Creating checkpoint directory: /tmp/sparta/checkpoint/kakfa_cabinfo
09 May 2017 07:48:51.389 INFO c.s.s.d.s.StreamingContextService Checkpoint created in folder: /tmp/sparta/checkpoint/kakfa_cabinfo
09 May 2017 07:48:51.389 INFO c.s.s.s.core.helpers.PolicyHelper$ Initializing reflection
09 May 2017 07:48:51.443 INFO c.s.sparta.driver.SpartaWorkflow Clearing last error for context: aa4d828e-5d2a-4ed0-9dc7-784b0bdef43d
09 May 2017 07:48:51.444 INFO c.s.sparta.driver.SpartaWorkflow Output: parquet_output created correctly.
09 May 2017 07:48:51.445 INFO c.s.sparta.driver.SpartaWorkflow Input: kafka-krb created correctly.
09 May 2017 07:48:51.446 INFO c.s.sparta.driver.SpartaWorkflow Stream for Input: kafka-krb created correctly.
09 May 2017 07:48:51.446 INFO c.s.sparta.driver.SpartaWorkflow RawData created correctly.
09 May 2017 07:48:51.447 INFO c.s.sparta.driver.SpartaWorkflow RawData: created correctly.
09 May 2017 07:48:51.449 ERROR c.s.sparta.driver.SpartaWorkflow An error was detected : PolicyErrorModel(Something gone wrong creating the parser: Ingestion. xer,The component couldn't be found in classpath. Please check the type.,Tue May 09 07:48:51 UTC 2017)
09 May 2017 07:48:51.450 INFO c.s.sparta.driver.SpartaWorkflow Updating context aa4d828e-5d2a-4ed0-9dc7-784b0bdef43d with name kakfa_cabinfo:
Status: NotStarted ---> NotStarted
Status Information: Starting Sparta local job for policy ---> Starting Sparta local job for policy
Submission Id: undefined ---> undefined
Submission Status: undefined ---> undefined
Marathon Id: undefined ---> undefined
Last Error: undefined ---> PolicyErrorModel(Something gone wrong creating the parser: Ingestion. Please re-check the policy.,Parser,The component couldn't be found in classpath. Please check the type.,Tue May 09 07:48:51 UTC 2017)
Last Execution Mode: local ---> local
Resource Manager URL: undefined ---> undefined
09 May 2017 07:48:51.453 ERROR c.s.s.s.a.actor.LocalLauncherActor Error initiating Sparta local job
java.lang.IllegalArgumentException: Something gone wrong creating the parser: Ingestion. Please re-check the policy.
at com.stratio.sparta.driver.stage.BaseStage$class.logAndCreateEx(BaseStage.scala:80)
at com.stratio.sparta.driver.SpartaWorkflow.logAndCreateEx(SpartaWorkflow.scala:31)
at com.stratio.sparta.driver.stage.BaseStage$class.generalTransformation(BaseStage.scala:59)
at com.stratio.sparta.driver.SpartaWorkflow.generalTransformation(SpartaWorkflow.scala:31)
at com.stratio.sparta.driver.stage.ParserStage$class.createParser(ParserStage.scala:54)
at com.stratio.sparta.driver.SpartaWorkflow.createParser(SpartaWorkflow.scala:31)
at com.stratio.sparta.driver.stage.ParserStage$$anonfun$parserStage$1.apply(ParserStage.scala:38)
at com.stratio.sparta.driver.stage.ParserStage$$anonfun$parserStage$1.apply(ParserStage.scala:38)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.immutable.List.map(List.scala:285)
at com.stratio.sparta.driver.stage.ParserStage$class.parserStage(ParserStage.scala:38)
at com.stratio.sparta.driver.SpartaWorkflow.parserStage(SpartaWorkflow.scala:31)
at com.stratio.sparta.driver.SpartaWorkflow$$anonfun$run$2.apply(SpartaWorkflow.scala:53)
at com.stratio.sparta.driver.SpartaWorkflow$$anonfun$run$2.apply(SpartaWorkflow.scala:50)
at scala.Option.foreach(Option.scala:257)
at com.stratio.sparta.driver.SpartaWorkflow.run(SpartaWorkflow.scala:50)
at com.stratio.sparta.driver.service.StreamingContextService.localStreamingContext(StreamingContextService.scala:53)
at com.stratio.sparta.serving.api.actor.LocalLauncherActor$$anonfun$1.apply$mcV$sp(LocalLauncherActor.scala:58)
at com.stratio.sparta.serving.api.actor.LocalLauncherActor$$anonfun$1.apply(LocalLauncherActor.scala:49)
at com.stratio.sparta.serving.api.actor.LocalLauncherActor$$anonfun$1.apply(LocalLauncherActor.scala:49)
at scala.util.Try$.apply(Try.scala:192)
at com.stratio.sparta.serving.api.actor.LocalLauncherActor.com$stratio$sparta$serving$api$actor$LocalLauncherActor$$doInitSpartaContext(LocalLauncherActor.scala:49)
at com.stratio.sparta.serving.api.actor.LocalLauncherActor$$anonfun$receive$1.applyOrElse(LocalLauncherActor.scala:40)
at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
at com.stratio.sparta.serving.api.actor.LocalLauncherActor.aroundReceive(LocalLauncherActor.scala:34)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: com.stratio.sparta.serving.core.exception.ServingCoreException: Class with name IngestionParser Cannot be found in the classpath.
at com.stratio.sparta.serving.core.exception.ServingCoreException$.create(ServingCoreException.scala:24)
at com.stratio.sparta.serving.core.utils.ReflectionUtils.tryToInstantiate(ReflectionUtils.scala:42)
at com.stratio.sparta.driver.stage.ParserStage$$anonfun$createParser$1.apply(ParserStage.scala:57)
at com.stratio.sparta.driver.stage.ParserStage$$anonfun$createParser$1.apply(ParserStage.scala:54)
at scala.util.Try$.apply(Try.scala:192)
at com.stratio.sparta.driver.stage.BaseStage$class.generalTransformation(BaseStage.scala:55)
... 34 common frames omitted
Caused by: java.lang.ClassNotFoundException: IngestionParser
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.stratio.sparta.serving.core.utils.ReflectionUtils.tryToInstantiate(ReflectionUtils.scala:38)
... 38 common frames omitted

@compae
Copy link
Member

compae commented May 11, 2017

Hi,

could you paste your reference.conf?

Thanks.

@vardancse
Copy link
Author

root@sparta1:/etc/sds/sparta/marathon# cat reference.conf

# AKKA
akka.log-dead-letters = off
akka.loggers = ["akka.event.slf4j.Slf4jLogger"]

@compae
Copy link
Member

compae commented May 15, 2017

And the file in /etc/sds/sparta/reference.conf?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants