Implement a utility tool for syncing parquet files to all three formats delta, hudi, iceberg using file listing

[vinishjail97]

Feature Request / Improvement

Sub-task of main feature request #550

Look at Using List files in the approach mentioned below. It's only a high level approach, PR author can change it and improve it as well.
#550 (comment)

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[sudharshanraja-db]

Hi @vinishjail97 i am working on this task (last week was on and off with this) should be able to submit PR by next week just wanted to share the update here

[vinishjail97]

okay, you can submit a draft PR as well if you are blocked on anything.

[sudharshanraja-db]

sure will do that

[soumilshah1995]

+1

[lalit2001]

is there any update on this ??

[sudhar91]

Hi @vinishjail97
I made the initial commit and raised draft PR while i am continuing to refactor the code raised PR to get some feedback around approach. requesting your review here
I was primarily validating against delta due to familiarity but will continue to test for other formats as well.
I intentionally didn't checkin the tests as i have started to see some weird jar conflicts will push once am done but i was building running on my local system to test

TODO in my list

1. Improve exception handling
2. Improve doc strings and update documentation
3. Simplify Certain logic
4. Consistency in column stats population
5. Test different scenarios of parquet ( multiple partitions..etc)
6. Add more coverage

Your review comments and suggestion would be helpful as i make changes further