[feature](backup)(cooldown) backup/restore for cooldown data #43642

justfortaste · 2024-11-11T15:27:57Z

What problem does this PR solve?

https://doc.weixin.qq.com/doc/w3_AGkArQbjACc5rBcdFOxQcaBSzESxW?scode=AJEAIQdfAAoRB9aawYAGkArQbjACc
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Currently, backup/resore is not supported for cooldown data.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.
  Feature backup restore cooldown data doris-website#1336

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-11T15:28:02Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

github-actions

clang-tidy made some suggestions

be/src/olap/rowset/beta_rowset.cpp

justfortaste · 2024-11-12T03:59:09Z

run buildall

fe/fe-core/src/main/java/org/apache/doris/analysis/RestoreStmt.java

fe/fe-core/src/main/java/org/apache/doris/backup/BackupMeta.java

fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java

fe/fe-core/src/main/java/org/apache/doris/catalog/PartitionInfo.java

fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java

be/src/olap/snapshot_manager.cpp

be/src/olap/rowset/beta_rowset.cpp

be/src/olap/tablet.cpp

be/src/runtime/snapshot_loader.cpp

xy720 · 2024-11-27T10:10:36Z

I suggest that in the first version, let the users specify the resource or policy in restore job, no need to recreate them.

fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java

be/src/runtime/snapshot_loader.cpp

xy720 · 2024-12-02T13:49:20Z

be/src/runtime/snapshot_loader.cpp

+    }
+
+    std::vector<std::string> remote_index_files;
+    RETURN_IF_ERROR(list_segment_inverted_index_file(cold_fs, remote_tablet_path, rowset_id,


May be we don't need to list remote files here. We can get index_id in rowset meta, index_version in tablet schema. And use remote_idx_v1_path/remote_idx_v2_path function in storage_policy.h to get the remote path.

be/src/runtime/snapshot_loader.cpp

justfortaste · 2024-12-05T00:28:32Z

run rebuildall

xy720

LGTM

xy720 · 2024-12-05T03:34:39Z

run buildall

github-actions · 2024-12-05T03:35:01Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-12-05T03:35:03Z

PR approved by anyone and no changes requested.

lide-reed

LGTM

doris-robot · 2024-12-05T04:43:00Z

TeamCity be ut coverage result:
Function Coverage: 38.46% (10006/26019)
Line Coverage: 29.49% (83861/284350)
Region Coverage: 28.60% (43105/150713)
Branch Coverage: 25.20% (21907/86948)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8258cf6d8c0ee77e312d0c99fdceda1b227d4135_8258cf6d8c0ee77e312d0c99fdceda1b227d4135/report/index.html

justfortaste · 2024-12-12T10:59:33Z

run buildall

github-actions · 2024-12-12T11:04:32Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-12T12:05:08Z

TeamCity be ut coverage result:
Function Coverage: 38.74% (10104/26080)
Line Coverage: 29.68% (84771/285649)
Region Coverage: 28.74% (43515/151400)
Branch Coverage: 25.30% (22107/87372)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8a6c45451ea029ab6fbed515ae40ea832f48e895_8a6c45451ea029ab6fbed515ae40ea832f48e895/report/index.html

justfortaste · 2024-12-14T12:16:53Z

run buildall

github-actions · 2024-12-14T12:22:14Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-14T13:31:39Z

TeamCity be ut coverage result:
Function Coverage: 38.74% (10107/26088)
Line Coverage: 29.68% (84856/285888)
Region Coverage: 28.75% (43555/151514)
Branch Coverage: 25.30% (22128/87456)
Coverage Report: http://coverage.selectdb-in.cc/coverage/328a6e5bab9045bfe4dbe511bf671477923c37f4_328a6e5bab9045bfe4dbe511bf671477923c37f4/report/index.html

lide-reed

LGTM

xy720

LGTM

justfortaste · 2024-12-24T15:35:22Z

squash pr
#45589

dataroaring

It is a huge pr, we should review carefully.

There is some low level change, like abs_path = std::filesystem::path(fmt::format("s3://{}/{}", _bucket, _prefix)) / path;

I am not sure its impact.

justfortaste · 2024-12-26T04:19:45Z

It is a huge pr, we should review carefully.

There is some low level change, like abs_path = std::filesystem::path(fmt::format("s3://{}/{}", _bucket, _prefix)) / path;

I am not sure its impact.

RemoteFileSystem::download will call absolute_path two times

RemoteFileSystem::download(const Path& remote_file, const Path& local) {
     S3FileSystem::absolute_path // first ime
     S3FileSystem::download_impl
     FileSystem::file_size
         S3FileSystem::absolute_path // second time

when "path" is not absolue, the result will return with double "_prefix"

original code:

    Status absolute_path(const Path& path, Path& abs_path) const override {
        if (path.string().find("://") != std::string::npos) {
            // the path is with schema, which means this is a full path like:
            // s3://bucket/path/to/file.txt
            // so no need to concat with prefix
            abs_path = path;
        } else {
            // path with no schema
            abs_path = _prefix / path;
        }
        return Status::OK();
    }

This function is only effect with S3 file system, the path will be parsed by "S3URI::parse" to get the key.

github-actions bot reviewed Nov 11, 2024

View reviewed changes

be/src/olap/rowset/beta_rowset.cpp Outdated Show resolved Hide resolved

justfortaste mentioned this pull request Nov 11, 2024

[Feature] Support backup and restore cooldown tablet #43174

Open

3 tasks

justfortaste requested review from dataroaring, morningman, yiguolei and xiaokang as code owners November 18, 2024 09:38

justfortaste force-pushed the backup_cooldown_data_master branch from bf24504 to 99971a5 Compare November 18, 2024 13:25

xy720 reviewed Nov 27, 2024

View reviewed changes

be/src/runtime/snapshot_loader.cpp Outdated Show resolved Hide resolved

be/src/runtime/snapshot_loader.cpp Outdated Show resolved Hide resolved

be/src/runtime/snapshot_loader.cpp Outdated Show resolved Hide resolved

justfortaste force-pushed the backup_cooldown_data_master branch 3 times, most recently from 2fba7fa to 005403a Compare November 29, 2024 04:23

justfortaste requested a review from xy720 November 29, 2024 06:17

xy720 reviewed Dec 2, 2024

View reviewed changes

justfortaste requested a review from xy720 December 3, 2024 04:36

justfortaste force-pushed the backup_cooldown_data_master branch from 13a365e to 9e2f47b Compare December 4, 2024 06:06

xy720 previously approved these changes Dec 5, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 5, 2024

github-actions bot added the reviewed label Dec 5, 2024

lide-reed previously approved these changes Dec 5, 2024

View reviewed changes

justfortaste dismissed stale reviews from lide-reed and xy720 via 0db7dbb December 5, 2024 12:06

justfortaste added 11 commits December 12, 2024 18:57

fix tmp

f6cd077

update test_backup_restore_cold_data.groovy

afe66cc

use new storage policy id

74869af

update FeMetaVersion.VERSION_130

e2e5dd3

update for review

684c6cf

remove remote_file_info

75ff4b9

fix cooldown_meta_id

e2d619e

fix review2

90b1407

fix local_files_with_checksum

f6b6d85

fix testcase timeout

626b7d6

save storage policy to jobinfo instead of BackupMeta

8a6c454

justfortaste force-pushed the backup_cooldown_data_master branch from 796ed9b to 8a6c454 Compare December 12, 2024 10:59

Merge branch 'master' into backup_cooldown_data_master

328a6e5

justfortaste requested review from lide-reed and xy720 December 16, 2024 01:25

lide-reed approved these changes Dec 16, 2024

View reviewed changes

xy720 approved these changes Dec 16, 2024

View reviewed changes

xy720 mentioned this pull request Dec 18, 2024

[feature](cooldown)backup cooldown data #45589

Open

16 tasks

justfortaste closed this Dec 24, 2024

justfortaste reopened this Dec 25, 2024

dataroaring requested changes Dec 25, 2024

View reviewed changes

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature](backup)(cooldown) backup/restore for cooldown data #43642

[feature](backup)(cooldown) backup/restore for cooldown data #43642

justfortaste commented Nov 11, 2024 •

edited

Loading

doris-robot commented Nov 11, 2024

github-actions bot left a comment

justfortaste commented Nov 12, 2024

xy720 commented Nov 27, 2024

xy720 Dec 2, 2024 •

edited

Loading

justfortaste commented Dec 5, 2024

xy720 left a comment

xy720 commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

lide-reed left a comment

doris-robot commented Dec 5, 2024

justfortaste commented Dec 12, 2024

github-actions bot commented Dec 12, 2024

doris-robot commented Dec 12, 2024

justfortaste commented Dec 14, 2024

github-actions bot commented Dec 14, 2024

doris-robot commented Dec 14, 2024

lide-reed left a comment

xy720 left a comment

justfortaste commented Dec 24, 2024

dataroaring left a comment

justfortaste commented Dec 26, 2024

[feature](backup)(cooldown) backup/restore for cooldown data #43642

Are you sure you want to change the base?

[feature](backup)(cooldown) backup/restore for cooldown data #43642

Conversation

justfortaste commented Nov 11, 2024 • edited Loading

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

doris-robot commented Nov 11, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

justfortaste commented Nov 12, 2024

xy720 commented Nov 27, 2024

xy720 Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

justfortaste commented Dec 5, 2024

xy720 left a comment

Choose a reason for hiding this comment

xy720 commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

lide-reed left a comment

Choose a reason for hiding this comment

doris-robot commented Dec 5, 2024

justfortaste commented Dec 12, 2024

github-actions bot commented Dec 12, 2024

doris-robot commented Dec 12, 2024

justfortaste commented Dec 14, 2024

github-actions bot commented Dec 14, 2024

doris-robot commented Dec 14, 2024

lide-reed left a comment

Choose a reason for hiding this comment

xy720 left a comment

Choose a reason for hiding this comment

justfortaste commented Dec 24, 2024

dataroaring left a comment

Choose a reason for hiding this comment

justfortaste commented Dec 26, 2024

justfortaste commented Nov 11, 2024 •

edited

Loading

xy720 Dec 2, 2024 •

edited

Loading