Start From LSN option? #263
tsumitobatsu
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi there. Firstly thank you for your work on this wonderful tool.
I see that table filtering has now been implemented to help reduce the amount of data retrieved where appropriate.
I have a use case where the same type of filtering would be of enormous benefit, based on a desired starting LSN value.
This builds upon the functionality already included, allowing the LSN to be retrieved along with the data such that a process can track its position in the WAL, but then allows that process to peek at rolling blocks of data by shifting its starting position each time.
As for the justifying use case, we would be using a single replication slot with multiple separate threads peeking at the data for different tables. Periodically the replication slot would have its starting LSN advanced to free up resources, but you might see how in such a setup each of those peeking threads would ideally need to be able to start streaming data from their last 'up-to' LSN+1 on each successive call.
Apologies if this seems eccentric, but we would find it quite a powerful feature actually. Equally, apologies if this is something which is already possible - e.g. via features built in to postgres itself - and I just missed them...
P.S. We're aware of pg_replication_slot_advance, but as the intention is to have multiple threads peeking at the data these could each reasonably have different LSN requirements on each request. Advancing the slot itself would therefore only be done periodically, more for resource management reasons.
Beta Was this translation helpful? Give feedback.
All reactions