[core] Introduce ExpireChangelogImpl to handle the changelog lifecycle #2942
Conversation
Aitozi
force-pushed
the
changelog-lifecycle
branch
from
da737dd to
b020e1f
Compare
| // move snapshot metadata to changelog | ||
| try { | ||
| snapshotManager.moveToChangelog(id); | ||
| snapshotManager.commitLongLivedChangelogLatestHint(id); |
There was a problem hiding this comment.
Maybe we should not just move snapshot file.
We could introduce a new JSON file for changelog.
There was a problem hiding this comment.
Currently, I've not seen the requirement to introduce a new metadata structure for changelog. We can rewrite the content later when we have to later.
There was a problem hiding this comment.
If we have to do it later, it is hard to keep forward compatibility.
It is better to provide a simple version now.
| snapshotManager.fileIO().deleteQuietly(snapshotManager.snapshotPath(id)); | ||
| // move snapshot metadata to changelog | ||
| try { | ||
| snapshotManager.moveToChangelog(id); |
There was a problem hiding this comment.
Please note that it is best not to affect the original behavior. For example, here we should determine whether there is a separate configuration for the changelog lifecycle. If it is not configured or is the same as the snapshot lifecycle, we should avoid creating files in the changelog folder.
| LOG.debug("Ready to delete changelog files from snapshot #" + id); | ||
| } | ||
| Snapshot snapshot = snapshotManager.snapshot(id); | ||
| if (snapshot.changelogManifestList() != null) { |
There was a problem hiding this comment.
If possible, we should try to ensure that the delta file of the Append snapshot can be stream read when there is no changelog file present.
There was a problem hiding this comment.
Agree, will give a try.
Purpose
Linked issue: close #2899
In this PR, the
snapshotwill be moved to the/changelogdirectory for decouple the changelog lifecycle. We will always keep one extra metadata file in the/changelogdirectory for the conveniency of streaming scan.Tests
API and Format
Documentation