Spark 4.0.0 thread

[alfonsorr]

I open this thread to keep track of the needs we find to prepare spark fast test to be usable in spark 4.0.0
Spark 4.0.0 is scala 2.13 compatible only. This will be the first time Spark uses this scala version as the main and only version. Previously it was compiled with scala 2.12 and 2.13 but only 2.12 was generally used in any production environment.

• Update sbt
• Prevent errors using new versions of Java #152
• Prepare build.sbt to accept 4.0.0
• Test against 4.0.0 and fix any errors that are found
  Optional but good to have:
• Get rid of all deprecation warnings working with scala 2.13 without breaking 2.12 compatibility

This block will be updated with any new task found

[alfonsorr]

I already have a branch that allows to use of the Spark 4 snapshot versions. I can compile, but I'm getting an error when testing. I will upload it as a draft when it's working.
I will work in an additional branch on the deprecation warnings.

Edit: Spark 4 is compatible with Java 17 and 21, I was working with java 8 to have the same version as databricks 😓