Spark 3.20 <-> Scala 2.13

[ex0ns]

Spark 3.2.0 was released on October 20th, it's the first release of spark with support for Scala 2.13, is there any plan to support this version, if yes what would be the timeline ?

The first issue that comes to my mind is related to the upgrade to spark-avro 3.2.0, I've given this a try, and after a few changes:

diff --git a/pom.xml b/pom.xml
index a782ddd..800f283 100644
--- a/pom.xml
+++ b/pom.xml
@@ -51,6 +51,7 @@
         <spark-23.version>2.3.4</spark-23.version>
         <spark-24.version>2.4.6</spark-24.version>
         <spark-30.version>3.0.0</spark-30.version>
+        <spark-32.version>3.2.0</spark-32.version>
 
         <spark.version>${spark-24.version}</spark.version>
 
@@ -61,6 +62,7 @@
 
         <scala_2.11.version>2.11.12</scala_2.11.version>
         <scala_2.12.version>2.12.10</scala_2.12.version>
+        <scala_2.13.version>2.13.6</scala_2.13.version>
 
         <!-- Controlled by `scala-cross-build` plugin -->
         <scala.version>2.12.10</scala.version>
@@ -79,6 +81,7 @@
 
         <!--Libs-->
         <spark.avro.version>2.4.6</spark.avro.version>
+        <spark-32.avro.version>3.2.0</spark-32.avro.version>
         <avro.version>1.8.2</avro.version>
 
         <scm.url><!-- defined outside the pom --></scm.url>
@@ -323,6 +326,13 @@
                 <scala.version>${scala_2.12.version}</scala.version>
             </properties>
         </profile>
+        <profile>
+            <id>scala-2.13</id>
+            <properties>
+                <scala.compat.version>2.13</scala.compat.version>
+                <scala.version>${scala_2.13.version}</scala.version>
+            </properties>
+        </profile>
 
         <!-- Spark profiles -->
         <profile>
@@ -343,6 +353,13 @@
                 <spark.version>${spark-30.version}</spark.version>
             </properties>
         </profile>
+        <profile>
+            <id>spark-3.2</id>
+            <properties>
+                <spark.version>${spark-32.version}</spark.version>
+                <spark.avro.version>${spark-32.avro.version}</spark.avro.version>
+            </properties>
+        </profile>
 
         <profile>
             <id>uber</id>

Thanks

[cerveada]

Yes, if everything goes right Abris 6 (that will support Spark 3.2 and Scala 2.12 and 2.13.) should be released in a few weeks.
Abris 5 will still be supported, and it will stay the go-to version for Spark 3.1 and older.