Pghoard restore_command optimization #618
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| os.unlink(prefetch_target_path) | ||
| return e | ||
| with self.server.lock: | ||
| os.rename(src, dst) |
Check failure
Code scanning / CodeQL
Uncontrolled data used in path expression High
| os.unlink(prefetch_target_path) | ||
| return e | ||
| with self.server.lock: | ||
| os.rename(src, dst) |
Check failure
Code scanning / CodeQL
Uncontrolled data used in path expression High
| @@ -443,6 +450,8 @@ | |||
| xlog_file = Path(target_path).resolve() | |||
| if data_dir not in xlog_file.parents: | |||
| raise HttpResponse(f"Invalid xlog file path {target_path}, it should be in data directory", status=400) | |||
| if not xlog_file.parent.is_dir(): | |||
Check failure
Code scanning / CodeQL
Uncontrolled data used in path expression High
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #618 +/- ##
==========================================
- Coverage 91.01% 90.79% -0.22%
==========================================
Files 31 31
Lines 4917 4954 +37
==========================================
+ Hits 4475 4498 +23
- Misses 442 456 +14
|
egor-voynov-aiven
force-pushed
the
egor-voynov-improve-restore-command
branch
2 times, most recently
from
a36a724 to
ebe7234
Compare
egor-voynov-aiven
force-pushed
the
egor-voynov-improve-restore-command
branch
3 times, most recently
from
23db920 to
92b17da
Compare
egor-voynov-aiven
force-pushed
the
egor-voynov-improve-restore-command
branch
from
92b17da to
d03e403
Compare
0xlianhu
reviewed
Apr 2, 2024
0xlianhu
reviewed
Apr 2, 2024
0xlianhu
reviewed
Apr 2, 2024
0xlianhu
previously requested changes
Apr 2, 2024
0xlianhu
reviewed
Apr 2, 2024
egor-voynov-aiven
force-pushed
the
egor-voynov-improve-restore-command
branch
4 times, most recently
from
a05baab to
6a53f07
Compare
Added a new DownloadResultsProcessor thread, which validates downloaded files and saves them to the target: '<pg_wal>/<wal_name>.tmp' -> '<pg_wal>/<wal_name>.prefetch'. Files with "prefetch" suffix can be copied to the destination without extra checks now. Also changed items type for "pending_download_ops" dict from "dict" to "dataclass" (PendingDownloadOp) Fixed typing issues for webserver.py renaming prefetched WAL in python CLI "pghoard_postgres_command" renaming prefetched WAL in "pghoard_postgres_command_go"
egor-voynov-aiven
force-pushed
the
egor-voynov-improve-restore-command
branch
from
30eb4e5 to
7c466c3
Compare
0xlianhu
reviewed
Apr 11, 2024
0xlianhu
reviewed
Apr 11, 2024
egor-voynov-aiven
force-pushed
the
egor-voynov-improve-restore-command
branch
from
63d2b2f to
2f6cfc2
Compare
egor-voynov-aiven
force-pushed
the
egor-voynov-improve-restore-command
branch
from
2f6cfc2 to
c367c2c
Compare
0xlianhu
approved these changes
Apr 15, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes:
pg_wal/<wal_name>.tmp -> pg_wal/<wal_name>.pghoard.prefetch.Files with "prefetch" suffix can be copied to the destination without extra checks now.
pghoard_postgres_command_go[BF-2247]
We observed that the startup process was only using 60 to 80% of a CPU. This means that we’re not supplying WAL files fast enough or that the restore_command takes too long to complete.
The idea would be to bypass the HTTP-call needed to check if we have a WAL file present from the
restore_commandto the pghoard main process.