[Feature] Optimize the memory usage of PartitionExpire.

[liming30]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

1. When the partition expires, all ManifestEntry of the corresponding partition will be read into the memory, and then a new ManifestEntry whose FileKind is Delete will be generated. When the file of the expired partition is very large (hundreds of thousands), this will occupy a lot of memory.

2. In addition, when there may be conflicts in the commit, the files of the changed partitions will also be read into the memory for conflict checking.

Solution

We may not need to read it completely into memory, but return it in a stream.

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[xiangyuf]

@liming30 hi, could u pls assign this issue to me?

[JingsongLi]

Already optimized in the 0.8.0