spike mode for dynamic series adjustment #70
Conversation
|
Currently, the implementation alternates between the original series count and the spiked series count at each seriesChangeInterval. This approach was chosen for its simplicity. However, if we want to have the original series count for the first 3 minutes and then the spiked series count for the next 5 minutes, we could achieve this by adding a new flag. This would allow for more flexible and customized patterns for metrics generation. If this is not a concern, we can proceed with the current implementation. |
metrics/serve.go
Outdated
| if inSpike { | ||
| inSpike = false | ||
| *currentSeriesCount = initialSeriesCount | ||
| } else { | ||
| inSpike = true | ||
| *currentSeriesCount = int(float64(initialSeriesCount) * spikeMultiplier) | ||
| } |
There was a problem hiding this comment.
you could simplify this by just checking
if *currentSeriescount > initialSeriesCount {
set to initial series count
continue
}
multiply by the spike multiplier
| select { | ||
| case updateNotify <- struct{}{}: | ||
| default: |
There was a problem hiding this comment.
not sure what's happening here, and looking at the other handle functions we're doing the same, what do we need this select and the update notify channel for?
There was a problem hiding this comment.
the updateNotify channel are used here to:
Lines 140 to 149 in 4b732e1
Each time a notification is received, the metrics are updated and logged.
There was a problem hiding this comment.
You're right! We learned something. It looks like updateNotify is only used for the write mode, where avalanche is remote writing somewhere. So this isn't needed for what we're trying to do with prombench, but it is required in general 👍
|
might need to rebase or merge in master, looks like there might be a merge conflict? |
Signed-off-by: yomek33 <[email protected]>
Signed-off-by: yomek33 <[email protected]>
@cstyan I fixed!🙆♀️ |
yomek33
force-pushed
the
series-spike
branch
3 times, most recently
from
69d0f7d to
98823c6
Compare
Signed-off-by: yomek33 <[email protected]>
metrics/serve_test.go
Outdated
| assert.Equal(t, expectedCount, currentCount, "Halved series count should be %d but got %d", int(expectedCount), currentCount) | ||
| } else { | ||
| currentCount := countSeries(t, promRegistry) | ||
| expectedCount := initialSeriesCount * spikeMultiplier | ||
| assert.Equal(t, int(expectedCount), currentCount, "Doubled series count should be %d but got %d", int(expectedCount), float64(currentCount)) |
There was a problem hiding this comment.
we should change the messages here, you could put "multiplied the series count by %d, should be %d but got %d" for example
| @@ -91,7 +99,7 @@ func main() { | |||
|
|
|||
| stop := make(chan struct{}) | |||
| defer close(stop) | |||
| updateNotify, err := metrics.RunMetrics(*metricCount, *labelCount, *seriesCount, *seriesChangeRate, *maxSeriesCount, *minSeriesCount, *metricLength, *labelLength, *valueInterval, *labelInterval, *metricInterval, *seriesChangeInterval, *seriesOperationMode, *constLabels, stop) | |||
| updateNotify, err := metrics.RunMetrics(*metricCount, *labelCount, *seriesCount, *seriesChangeRate, *maxSeriesCount, *minSeriesCount, *metricLength, *labelLength, *valueInterval, *labelInterval, *metricInterval, *seriesChangeInterval, *spikeMultiplier, *seriesOperationMode, *constLabels, stop) | |||
There was a problem hiding this comment.
We don't need to do it now, but we should look at breaking RunMetrics into separate functions for each of the run modes. The current functions such as this new handleSpikeMode could create the other go routines that we create within RunMetrics currently.
This would mean we don't have to pass so many parameters around across a few functions.
Signed-off-by: yomek33 <[email protected]>
Add spike mode for metrics generation and cycling. Other two modes: #64
spike
usage
./avalanche --series-operation-mode=spike --series-change-interval=180 --series-count=100 --spike-multiplier=1.5This mode periodically increases the series count by a spike multiplier on one tick and then returns it to the original count on the next tick. This pattern repeats indefinitely.