storage: make probe intervals runtime configurable #31125
+285
−107
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
teskje
force-pushed
the
dynamic-upstream-fetch-intervals
branch
from
b9884a2 to
4348ff6
Compare
This commit ensures that probe timestamps, and therefore timestamps used for minting new bindings, are rounded down to the probe interval (typically 1s). This is to reduce the amount of distinct timestamps flowing through a dataflow joining multiple sources. Each distinct timestamp induces some amount of overhead, so forcing the timestamps of individual sources to the same values is more efficient.
This commit removes the `KAFKA_DEFAULT_METADATA_FETCH_INTERVAL` legacy
config and replaces it with a `KAFKA_METADATA_FETCH_INTERVAL` dyncfg.
Like `{PG,MYSQL}_OFFSET_KNOWN_INTERVAL`, this makes it possible to
dynamically observe updates to this interval through a `ConfigSet`,
apart from removing some boilerplate.
The flag also loses its "default" part and is now the only thing
determining the probe ticker interval. This is the first step in
removing the user-visible option to configure the Kafka source metadata
fetch interval and is done here because it simplifies dynamically
updating the probe ticker interval in the next commit.
This commit makes it possible to change the tick interval of the upstream frontier probes without a source restart. To this end, the `probe::Ticker` receives a closure with which it can request the current interval, which it calls between ticks. The various source implementations implement the closure to read their respective dyncfgs.
teskje
force-pushed
the
dynamic-upstream-fetch-intervals
branch
from
4348ff6 to
02a39cd
Compare
antiguru
approved these changes
Jan 22, 2025
| @@ -18,7 +20,6 @@ import "service/src/params.proto"; | |||
| import "tracing/src/params.proto"; | |||
|
|
|||
| message ProtoStorageParameters { | |||
| reserved 1, 2, 3, 25, 12, 16, 26, 31, 39; | |||
There was a problem hiding this comment.
We could re-use the small identifiers, they have a more efficient proto encoding.
There was a problem hiding this comment.
Good point, might as well clean up the ids here too.
There was a problem hiding this comment.
I've done that and also reordered the proto fields to match the order in Rust.
The protobuf types in this file are only used in the storage protocol and never used across Mz versions.
teskje
force-pushed
the
dynamic-upstream-fetch-intervals
branch
from
02a39cd to
2ba022c
Compare
aljoscha
approved these changes
Jan 22, 2025
test/testdrive/source-timestamps.td
Outdated
| @@ -59,9 +59,6 @@ ALTER SYSTEM SET kafka_default_metadata_fetch_interval = '1s' | |||
| ALTER SYSTEM SET pg_offset_known_interval = '1s' | |||
| ALTER SYSTEM SET mysql_offset_known_interval = '1s' | |||
|
|
|||
| > ALTER CLUSTER test SET (REPLICATION FACTOR 0) | |||
There was a problem hiding this comment.
mostly just curious: why are these removed?
There was a problem hiding this comment.
They existed because applying the changed intervals needed a source restart before this PR, but it doesn't need one now!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR makes it possible to change the tick interval of the upstream frontier probes without a source restart.
To simplify things I've added a first (currently second) commit that moves the Kafka interval configuration into dyncfg and starts ignoring the user-defined per-source interval (which is to be removed in https://github.com/MaterializeInc/database-issues/issues/8887).
The first commit is from #31072 and can be ignored here.
Motivation
Closes https://github.com/MaterializeInc/database-issues/issues/8886
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.