[spark] remove the dependency of paimon-bundle in all submodules of paimon-spark.

[liming30]

Purpose

Linked issue: close #4351

Remove the dependency of paimon-bundle in all submodules of paimon-spark to fix the problem that the paimon class cannot be found in idea.

Tests

API and Format

Documentation

[wwj6591812]

+1

[liming30]

@JingsongLi @Zouxxyy can I merge this PR?

[Zouxxyy]

@JingsongLi @Zouxxyy can I merge this PR?

Sorry for delay, let me test it on my idea 2024

[Zouxxyy]

I test MigrateFileProcedureTest on my IDEA and it throw this:

An exception or error caused a run to abort: COMPRESSION_ZSTD_LEVEL 
java.lang.NoSuchFieldError: COMPRESSION_ZSTD_LEVEL
	at org.apache.paimon.format.orc.OrcFileFormat.getOrcProperties(OrcFileFormat.java:152)
	at org.apache.paimon.format.orc.OrcFileFormat.<init>(OrcFileFormat.java:75)
	at org.apache.paimon.format.orc.OrcFileFormatFactory.create(OrcFileFormatFactory.java:35)
	at org.apache.paimon.format.orc.OrcFileFormatFactory.create(OrcFileFormatFactory.java:24)
	at org.apache.paimon.format.FileFormat.fromIdentifier(FileFormat.java:106)
	at org.apache.paimon.format.FileFormat.fromIdentifier(FileFormat.java:91)

[liming30]

I test MigrateFileProcedureTest on my IDEA and it throw this:

An exception or error caused a run to abort: COMPRESSION_ZSTD_LEVEL 
java.lang.NoSuchFieldError: COMPRESSION_ZSTD_LEVEL
	at org.apache.paimon.format.orc.OrcFileFormat.getOrcProperties(OrcFileFormat.java:152)
	at org.apache.paimon.format.orc.OrcFileFormat.<init>(OrcFileFormat.java:75)
	at org.apache.paimon.format.orc.OrcFileFormatFactory.create(OrcFileFormatFactory.java:35)
	at org.apache.paimon.format.orc.OrcFileFormatFactory.create(OrcFileFormatFactory.java:24)
	at org.apache.paimon.format.FileFormat.fromIdentifier(FileFormat.java:106)
	at org.apache.paimon.format.FileFormat.fromIdentifier(FileFormat.java:91)

@Zouxxyy There is a version conflict with orc-core. When I exclude it, all the tests in idea can run successfully. But when I use mvn to execute in the terminal, an exception occurs that orc classes cannot be found. Do you have any other suggestions?

[Zouxxyy]

@Zouxxyy There is a version conflict with orc-core. When I exclude it, all the tests in idea can run successfully. But when I use mvn to execute in the terminal, an exception occurs that orc classes cannot be found. Do you have any other suggestions?

@liming30 In fact, I have no other suitable solutions yet. Can you try 1) remove paimon-bundle, 2) exclude orc or parquet deps in spark deps 3) add orc or parquet deps with paimon used version

[Zouxxyy]

@liming30 Can you try this Zouxxyy@05d3a86

[liming30]

@liming30 Can you try this Zouxxyy@05d3a86

Thanks for your help, I'll try this again later.

[liming30]

@Zouxxyy There are still some tests that fail in idea, such as org.apache.paimon.spark.procedure.MigrateTableProcedureTest. I will try to solve this problem next.

[askwang]

@Zouxxyy There are still some tests that fail in idea, such as org.apache.paimon.spark.procedure.MigrateTableProcedureTest. I will try to solve this problem next.

@liming30 I tested it ok in my idea, maybe you need rebuild the project

[YannByron]

That works. +1.