Minimal contributing guide #42
Conversation
- guide - fix & update links
| @@ -3,7 +3,7 @@ | |||
| A modern PySpark Connect/Classic wrapper on top of the Deequ, a beautiful Data Quality library from AWS Labs. | |||
|
|
|||
|
|
|||
|  | |||
|  | |||
|
|
|||
| **_NOTE:_** _Tsumugi Shiraui is a chimera: a hybrid of Human and Gauna. She combines the chaotic power of Gauna with a Human intillegence and empathia. Like an original character of the Manga "Knights of Sidonia", this project aims to make a hybrid of very powerful but hard to learn and use Deequ Scala Library with a usability and simplicity of Spark Connect (PySpark Connect, Spark Connect Go, Spark Connect Rust, etc.)._ | |||
|
|
|||
There was a problem hiding this comment.
Not related to this PR but On line 12 there is a typo Lungauge can we include it in this PR
|
|
||
| ## Running examples or testing clients | ||
|
|
||
| To simplify testing and development, there is a script that builds a server plugin, downloads and unpacks the Spark distribution, and runs the Spark Connect Server with all the necessary configurations. To run it, use `make run_spark_server`. After that, the server will be available at `sc://localhost:15002`. |
There was a problem hiding this comment.
make run_spark_server results in a build error - [ERROR] /workspaces/tsumugi-spark/tsumugi-server/src/test/scala/com/ssinchenko/tsumugi/DeequUtilsTest.scala:66: type mismatch; found : scala.util.Try[com.amazon.deequ.VerificationRunBuilder] required: com.amazon.deequ.VerificationRunBuilder [ERROR] one error found
Changing the statement val deequSuite = DeequSuiteBuilder.protoToVerificationSuite(data, protoSuiteBuilder.build()) to val deequSuite = DeequSuiteBuilder.protoToVerificationSuite(data, protoSuiteBuilder.build()).getOrElse(throw new RuntimeException("Failed to create VerificationSuite")) in /workspaces/tsumugi-spark/tsumugi-server/src/test/scala/com/ssinchenko/tsumugi/DeequUtilsTest.scala and adding relevant imports resolved it, however my Scala is no good. Please have a look and use your judgement!
There was a problem hiding this comment.
Good catch. I just realized that I somehow missed this in previous PR and we actually do not have automated testing for scala server. Will create a separate Issue to tackle this. Since it's not really related to this PR I will merge it in
There was a problem hiding this comment.
I will try to fix the problem in this PR but we also need to run some minimal python client tests in CI
There was a problem hiding this comment.
@SemyonSinchenko Sorry Did not get updated on this. I figured we already have #40 which might be a good candidate for this?
Close #37