spark-bigtable-connector: Scala Version Upgrade (2.12 -> 2.13) #52
Conversation
- module restructure (separate module for scala2.12 and scala2.13) - third_party folder is restructured: separate folders for SchemaConverters class are created (scala2.12 and scala2.13) - common spark bigtable source code is kept under spark-bigtable-core folder (pom.xml is removed as this source files will be used both in scala2.12 and scala2.13 with third_party source code)
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
| @@ -56,8 +56,12 @@ object WordCount extends App { | |||
| ) | |||
| .drop("frequency_double") | |||
|
|
|||
| // "bigtable" is not working - throwing Data Source not found | |||
| // when full path like below is provided, it works! | |||
| val bigtableFormat = "com.google.cloud.spark.bigtable.BigtableDefaultSource" | |||
There was a problem hiding this comment.
Does this work without this PR's changes? If so this would be a breaking change and we should try to understand why it is failing
There was a problem hiding this comment.
Resolved. Now it is working with "bigtable"
spark-bigtable-scala2.12/pom.xml
Outdated
There was a problem hiding this comment.
Please add a license header (like the one on https://github.com/GoogleCloudDataproc/spark-bigtable-connector/blob/main/pom.xml)
spark-bigtable-scala2.13/pom.xml
Outdated
There was a problem hiding this comment.
Please add a license header (like the one on https://github.com/GoogleCloudDataproc/spark-bigtable-connector/blob/main/pom.xml)
pom.xml
Outdated
| @@ -94,6 +94,198 @@ | |||
| <commons-lang.version>2.6</commons-lang.version> | |||
| <openlineage.version>1.22.0</openlineage.version> | |||
| </properties> | |||
| <dependencies> | |||
There was a problem hiding this comment.
Should some (all?) of these dependencies be on the -core package? Things like google-cloud-bigtable and such are dependencies for core, but since the parent package doesn't really have any source code we can leave these out of here and keep this pom focused on package assembly
There was a problem hiding this comment.
core is just to provide all the source file. Created a new module (spark-bigtable). scala2.12 and scala2.13 are placed inside it. common dependencies are part of this new module now.
- created a main module and common dependency are moved from root pom to spark-bigtable/pom
- spark.read.format("bigtable") is working properly now
parent PR - #51