Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace Gzip with PGzip #266

Merged
merged 7 commits into from
Sep 3, 2023
Merged

Replace Gzip with PGzip #266

merged 7 commits into from
Sep 3, 2023

Conversation

MaxJa4
Copy link
Contributor

@MaxJa4 MaxJa4 commented Sep 2, 2023

Closes #265

Replaces compress/gzip with github.com/klauspost/pgzip (drop-in replacement, compatible) to significantly increase compression (archiving) performance (1.5-8+ times depending on setting GZIP_PARALLELISM).
Default is 1 to not use up all available compute capacity on production servers during the archiving process.


@m90: Also added the --timeout 3 for docker compose down commands in the test suite.

@MaxJa4
Copy link
Contributor Author

MaxJa4 commented Sep 2, 2023

New test fails with /bin/sh: /code/test/pgzip/run.sh: Permission denied right at the beginning, without even executing the test. It works locally. Do you know what is happening here @m90?

@m90
Copy link
Member

m90 commented Sep 3, 2023

Sorry for not mentioning this before, the run.sh file needs to be executable, i.e. you need to chmod +x run.sh still.

cmd/backup/archive.go Outdated Show resolved Hide resolved
cmd/backup/config.go Outdated Show resolved Hide resolved
@MaxJa4
Copy link
Contributor Author

MaxJa4 commented Sep 3, 2023

the run.sh file needs to be executable

It was executable by default locally, but I had to run git add --chmod=+x run.sh to make git aware of that. Additionally GitHub Desktop sees a change after that, but when committing it says "error: no changes". Doing everything with command line and VS Code worked though.

@m90
Copy link
Member

m90 commented Sep 3, 2023

but I had to run git add --chmod=+x run.sh to make git aware of that

Strange, usually git should pick this up all by itself. Could it be that you set core.filemode to false somehow?

Anyways, good to see it's running as expected now.

cmd/backup/config.go Outdated Show resolved Hide resolved
cmd/backup/script.go Outdated Show resolved Hide resolved
@MaxJa4
Copy link
Contributor Author

MaxJa4 commented Sep 3, 2023

I renamed WholeNumber again as its logically false. The name suggests that any whole number (including negative ones) are allowed - which is not the case. Since its exactly like NaturalNumber but with a zero, I just named it NaturalNumber0 now to align it a little with the mathematical convention. I also added a doc string for both types to avoid confusion.

@m90
Copy link
Member

m90 commented Sep 3, 2023

Hmm, I got all my knowledge from Wikipedia which says whole numbers are natural numbers + 0. Then again, it's probably disputed and not really well defined.

@MaxJa4
Copy link
Contributor Author

MaxJa4 commented Sep 3, 2023

Could it be that you set core.filemode to false somehow?

Not that I know of. I just changed to true, but now GIT says I have changes in all .sh files... so it definitely picked something up and was turned off before.

which says whole numbers are natural numbers + 0

Seems like I found exactly the opposite side of that discussion haha. Well, I don't have a real preference here, now that the doc string is there to explain it.

test/util.sh Show resolved Hide resolved
Copy link
Member

@m90 m90 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚄

@m90 m90 merged commit 336c5be into offen:main Sep 3, 2023
2 checks passed
@MaxJa4 MaxJa4 deleted the compression-pgzip branch September 3, 2023 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use more efficient drop-in replacements for compression
2 participants