Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add -once and -force-snapshot arguments to replicate command #486

Open
hifi opened this issue Jun 13, 2023 · 1 comment · May be fixed by #488
Open

Add -once and -force-snapshot arguments to replicate command #486

hifi opened this issue Jun 13, 2023 · 1 comment · May be fixed by #488

Comments

@hifi
Copy link
Collaborator

hifi commented Jun 13, 2023

Related to #436 I spoke with someone either in some issue or on Slack sometime ago that there's a need for Litestream to be able to be executed periodically instead of continuously.

Our use case is to ensure the local database has been fully replicated to remote storage before we remove it to prevent data loss when we are intentionally moving the process out from the host it was running on and starting it on another. The current workaround is to use exec but it's not always reliable enough and for scheduled maintenance it's sometimes a lot faster to upload a snapshot and download it with restore instead of using the WAL chunks.

My proposal is that the replicate command gains -once to make it only replicate anything it has not replicated before on startup if the WAL hasn't been rolled out by the application process and -force-snapshot to force a snapshot to be taken on startup regardless if -once was used or not. This allows us to use them in combination to do a final backup of a database and ensure it's consistent by running restore as well.

The -once argument needs force all the maintenance tasks as well or they need to be behind another flag like --force-retention

This change would also allow cron style executing of Litestream given two preconditions are met:

  • application uses PRAGMA wal_autocheckpoint = 0
  • application uses SQLITE_FCNTL_PERSIST_WAL
  • Litestream can issue a passive checkpoint during at least some of the runs

Of course in cron style execution the write pressure must be extremely low not to block Litestream checkpoints and to not grow the WAL ridiculously but for some workloads this might make sense - or they just want to take full snapshots once in a while and have a binary that can do encrypted uploads and downloads to a bucket and always get the latest one on restore with

I will implement something like this soon in downstream but it'd be great if I could architect it to be upstream compatible given it would be accepted in some form.
@hifi hifi mentioned this issue Jun 14, 2023
Closed
hifi added a commit to beeper/litestream that referenced this issue Jun 21, 2023
@hifi
Add replicate flags to pump Litestream manually
d71d7a4 
For some special setups it is sometimes useful to run Litestream
manually instead of letting it replicate in the background.

This commit implements the following flags for replicate:

  * -once for doing synchronous replication and then exit
  * -force-snapshot to force a snapshot during -once
  * -enforce-retention to enforce retention rules during -once

Because running once does not respect the snapshot interval the
caller is expected to use -force-snapshot and -enforce-retention
regularly to ensure the replication targets stay clean.

For this to work correctly with a live database it needs to be opened
with auto checkpointing disabled and SQLITE_FCNTL_PERSIST_WAL.

Other uses include only using -force-snapshot to create regular backups
of the database instead of live replication.

Fixes benbjohnson#486
hifi added a commit to beeper/litestream that referenced this issue Jun 21, 2023
@hifi
Add replicate flags to pump Litestream manually
072a755 
For some special setups it is sometimes useful to run Litestream
manually instead of letting it replicate in the background.

This commit implements the following flags for replicate:

  * -once for doing synchronous replication and then exit
  * -force-snapshot to force a snapshot during -once
  * -enforce-retention to enforce retention rules during -once

Because running once does not respect the snapshot interval the
caller is expected to use -force-snapshot and -enforce-retention
regularly to ensure the replication targets stay clean.

For this to work correctly with a live database it needs to be opened
with auto checkpointing disabled and SQLITE_FCNTL_PERSIST_WAL.

Other uses include only using -force-snapshot to create regular backups
of the database instead of live replication.

Fixes benbjohnson#486
@hifi hifi linked a pull request Jun 21, 2023 that will close this issue
Draft
@hifi
Copy link
Collaborator Author

hifi commented Jun 21, 2023

Created a draft PR. It sort of works and I'm interested in comments before going further with it.

hifi added a commit to beeper/litestream that referenced this issue Jun 21, 2023
@hifi
Add replicate flags to pump Litestream manually
fe5f922 
For some special setups it is sometimes useful to run Litestream
manually instead of letting it replicate in the background.

This commit implements the following flags for replicate:

  * -once for doing synchronous replication and then exit
  * -force-snapshot to force a snapshot during -once
  * -enforce-retention to enforce retention rules during -once

Because running once does not respect the snapshot interval the
caller is expected to use -force-snapshot and -enforce-retention
regularly to ensure the replication targets stay clean.

For this to work correctly with a live database it needs to be opened
with auto checkpointing disabled and SQLITE_FCNTL_PERSIST_WAL.

Other uses include only using -force-snapshot to create regular backups
of the database instead of live replication.

Fixes benbjohnson#486
hifi added a commit to beeper/litestream that referenced this issue Oct 17, 2023
@hifi
Add replicate flags to pump Litestream manually
41f0e90 
For some special setups it is sometimes useful to run Litestream
manually instead of letting it replicate in the background.

This commit implements the following flags for replicate:

  * -once for doing synchronous replication and then exit
  * -force-snapshot to force a snapshot during -once
  * -enforce-retention to enforce retention rules during -once

Because running once does not respect the snapshot interval the
caller is expected to use -force-snapshot and -enforce-retention
regularly to ensure the replication targets stay clean.

For this to work correctly with a live database it needs to be opened
with auto checkpointing disabled and SQLITE_FCNTL_PERSIST_WAL.

Other uses include only using -force-snapshot to create regular backups
of the database instead of live replication.

Fixes benbjohnson#486
hifi added a commit to beeper/litestream that referenced this issue Oct 20, 2023
@hifi
Add replicate flags to pump Litestream manually
ae9c078 
For some special setups it is sometimes useful to run Litestream
manually instead of letting it replicate in the background.

This commit implements the following flags for replicate:

  * -once for doing synchronous replication and then exit
  * -force-snapshot to force a snapshot during -once
  * -enforce-retention to enforce retention rules during -once

Because running once does not respect the snapshot interval the
caller is expected to use -force-snapshot and -enforce-retention
regularly to ensure the replication targets stay clean.

For this to work correctly with a live database it needs to be opened
with auto checkpointing disabled and SQLITE_FCNTL_PERSIST_WAL.

Other uses include only using -force-snapshot to create regular backups
of the database instead of live replication.

Fixes benbjohnson#486
hifi added a commit to beeper/litestream that referenced this issue Oct 20, 2023
@hifi
Add replicate flags to pump Litestream manually
9474f95 
For some special setups it is sometimes useful to run Litestream
manually instead of letting it replicate in the background.

This commit implements the following flags for replicate:

  * -once for doing synchronous replication and then exit
  * -force-snapshot to force a snapshot during -once
  * -enforce-retention to enforce retention rules during -once

Because running once does not respect the snapshot interval the
caller is expected to use -force-snapshot and -enforce-retention
regularly to ensure the replication targets stay clean.

For this to work correctly with a live database it needs to be opened
with auto checkpointing disabled and SQLITE_FCNTL_PERSIST_WAL.

Other uses include only using -force-snapshot to create regular backups
of the database instead of live replication.

Fixes benbjohnson#486
hifi added a commit to beeper/litestream that referenced this issue Oct 25, 2023
@hifi
Add replicate flags to pump Litestream manually
058c626 
For some special setups it is sometimes useful to run Litestream
manually instead of letting it replicate in the background.

This commit implements the following flags for replicate:

  * -once for doing synchronous replication and then exit
  * -force-snapshot to force a snapshot during -once
  * -enforce-retention to enforce retention rules during -once

Because running once does not respect the snapshot interval the
caller is expected to use -force-snapshot and -enforce-retention
regularly to ensure the replication targets stay clean.

For this to work correctly with a live database it needs to be opened
with auto checkpointing disabled and SQLITE_FCNTL_PERSIST_WAL.

Other uses include only using -force-snapshot to create regular backups
of the database instead of live replication.

Fixes benbjohnson#486
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add replicate flags to pump Litestream manually beeper/litestream
1 participant
@hifi