Skip to content

v0.0.84

Compare
Choose a tag to compare
@lyogev lyogev released this 25 May 11:25

New

  1. Added new mode called periodic, basically runs a batch job in a loop
  2. Selective merge UDF
  3. Added lag reporting to batch and streaming jobs
  4. Added empty output protection
  5. Support reading metrics from remote location (s3, HDFS)

Improvements

  1. Hudi: removeNullColumns flag to remove null columns before writing to hudi
  2. Hudi: deletePendingCompactions flag removes pending compactions when running in streaming
  3. Hudi: added a manual hive sync mode (this helps hive 1 users)
  4. Hudi: added hudiTableName parameter
  5. Hive: support hive 2.3
  6. Started releasing assembled JAR to maven central

Fixes

  1. Hudi: fix metrics reporting when using hudi in streaming
  2. Hudi: upgrade to 0.5.2 (breaking change)
  3. Refactored build process