relay splitter daemon + pebble persistence #810
Conversation
| binary.BigEndian.PutUint64(key[8:16], nowMillis) | ||
| binary.BigEndian.PutUint32(key[16:], pp.prevSeqExtra) | ||
|
|
||
| err = pp.db.Set(key[:], blob, pebble.Sync) |
There was a problem hiding this comment.
oh, we should just ignore -1 sequence number events (info, error, etc)
There was a problem hiding this comment.
I had it throwing away non-sequence events, but then figured out this clever way to include them in order
| Did string `gorm:"uniqueIndex"` | ||
| PDS uint | ||
| } | ||
| type User = pdsdata.User |
There was a problem hiding this comment.
feels like a superfluous change to include in this PR
There was a problem hiding this comment.
Yeah, but I was working on the unit tests, and the tests were in a different package which was weird, so I fixed that and then had to move a data definition to fix a circular dependency.
|
TODO: There's a pebble API "EstimateDiskUsage()", it would be fun to put that in a gauge. It's probably a little expensive so put that on a ~(1-5 minute) loop to set the gauge? Bonus round: use that to maintain a target size of the on disk buffer at e.g. 1TB rather than (N hours) |
|
TODO: I think we still have a TODO of using the pebble store's "get last entry" to set the sequence we fetch |
- limits on HTTP idle connections. mostly impacts did:web and HTTPS handle resolution, which can otherwise result in huge numbers of sockets hanging around (one per custom domain) when operating at scale. values chosen should not have too bad an impact on PLC usage, though we might want to split out separate PLC and non-PLC HTTP clients at some point - drop some timeouts
should cut ~3% of all DB usage on the prod relay
|
merged as #818 |
stack PR on top of #807
Add Pebble db based persister
clean up tests in events/