[Core][Flink][Spark] Add deletedFileTotalLenInBytes in result of OrphanFilesClean #4545
Conversation
wwj6591812
force-pushed
the
introduce_CleanOrphanFilesResult_for_support_deletedFileTotalSizeInBytes_1119
branch
from
10a4a45 to
94a868e
Compare
wwj6591812
force-pushed
the
introduce_CleanOrphanFilesResult_for_support_deletedFileTotalSizeInBytes_1119
branch
2 times, most recently
from
0361a11 to
da67df0
Compare
|
@JingsongLi |
wwj6591812
changed the title
| deleteFiles::add, | ||
| p -> { | ||
| try { | ||
| deletedFilesSizeInBytes.addAndGet(fileIO.getFileSize(p)); |
There was a problem hiding this comment.
getFileSize should always be together with delete, otherwise, I feel there may be issues with execution efficiency, getFileSize should be executed parallelismly too.
There was a problem hiding this comment.
getFileSize should always be together with delete, otherwise, I feel there may be issues with execution efficiency, getFileSize should be executed parallelismly too.
@JingsongLi Very thanks for your suggestions.
I would like to confirm with you the code improvement plan:
Are you means that getFileStatus execute only once in OrphanFilesClean#createFileCleaner, and fileCleaner can direct return the size of deleted file?
There was a problem hiding this comment.
I just think about it, maybe we don't need to invoke getFileSize, the files come from listStatus, it already contains file size.
There was a problem hiding this comment.
OK, Good idea.
I will have a try.
Thx
There was a problem hiding this comment.
I just think about it, maybe we don't need to invoke
getFileSize, the files come fromlistStatus, it already contains file size.
@JingsongLi
Hi，please cc, THX.
wwj6591812
force-pushed
the
introduce_CleanOrphanFilesResult_for_support_deletedFileTotalSizeInBytes_1119
branch
2 times, most recently
from
366f382 to
f97585e
Compare
wwj6591812
force-pushed
the
introduce_CleanOrphanFilesResult_for_support_deletedFileTotalSizeInBytes_1119
branch
from
f97585e to
66d1f25
Compare
wwj6591812
changed the title
Purpose
In my company, after we run a RemoveOrphanFilesAction, we not only want to know the number of orphan files that have been cleared, but also want to know how much capacity has been cleared.
This pr add deletedFileTotalLenInBytes in result of OrphanFilesClean.
Linked issue: close #xxx
Tests
API and Format
Documentation