Dynamic series adjustment #64

yomek33 · 2024-07-15T21:25:36Z

Add two new operation modes to the RunMetrics function: gradual-change and double-halve:

double-halve

usage
./avalanche --operation-mode=double-halve --series-change-interval=30 --series-count=20

This mode alternately doubles and halves the series count at regular intervals.
The series count is doubled on one tick and halved on the next, ensuring it never drops below 1.

gradual-change

usage
./avalanche --operation-mode=gradual-change --series-change-interval=30 --series-change-rate=10 series-count=20

This mode gradually changes the series count by a fixed rate at regular intervals
The series count is incremented by seriesChangeRate on each tick, ensuring it never drops below 1.

Tests

Added unit tests to cover the new double-halve and gradual-change modes.
Verified that the series count adjusts correctly and does not fall below 1.

cstyan

Looking good! Just a few minor implementation comments.

Looks like the tests are failing, there's a make target make test which should run them the same way as Circle CI is where they're failing.

cstyan · 2024-07-16T01:42:26Z

cmd/avalanche.go

+	labelInterval        = kingpin.Flag("series-interval", "Change series_id label values every {interval} seconds.").Default("60").Int()
+	metricInterval       = kingpin.Flag("metric-interval", "Change __name__ label values every {interval} seconds.").Default("120").Int()
+	seriesChangeInterval = kingpin.Flag("series-change-interval", "Change the number of series every {interval} seconds.").Default("10").Int()
+	operationMode        = kingpin.Flag("operation-mode", "Mode of operation: 'gradual-change', 'series-spike', 'double-halve'").Default("default").String()


series-spike doesn't look like it exists yet?

I will add spike mode in next PR! So, I deleted this part of text in this PR.

👍 sounds good

cstyan · 2024-07-16T01:50:26Z

metrics/serve.go

+	case "gradual-change":
+		go func() {
+			for tick := range changeSeriesTick.C {
+				fmt.Printf("%v: Adjusting series count. New count: %d\n", tick, seriesCount)
+				metricsMux.Lock()
+				seriesCount += seriesChangeRate
+				if seriesCount < 1 {
+					seriesCount = 1
+				}
+				unregisterMetrics()
+				registerMetrics(metricCount, metricLength, metricCycle, labelKeys)
+				cycleValues(labelKeys, labelValues, seriesCount, seriesCycle)
+				metricsMux.Unlock()
+
+				select {
+				case updateNotify <- struct{}{}:
+				default:
+				}
+			}
+		}()


one change I'd like to see to this functionality; it would be nice if we gradually increased up to some maximum, and then gradually decreased back down to the starting value (and repeat)

I have made the changes!

cstyan · 2024-07-16T01:52:31Z

metrics/serve.go

+	default:
+	}


I think the above go routines, before your switch statement, should maybe be the default case?

Otherwise we have both those AND the operation mode we've decided on both modifying the # of metrics and series.

I fixed🙆‍♀️

metrics/serve_test.go

cstyan · 2024-07-16T02:00:41Z

cmd/avalanche.go

+	metricCount          = kingpin.Flag("metric-count", "Number of metrics to serve.").Default("500").Int()
+	labelCount           = kingpin.Flag("label-count", "Number of labels per-metric.").Default("10").Int()
+	seriesCount          = kingpin.Flag("series-count", "Number of series per-metric.").Default("10").Int()
+	seriesChangeRate     = kingpin.Flag("series-change-rate", "The rate at which the number of active series changes over time. Positive value increases the series, negative value decreases the series.").Default("10").Int()


for the flags that only apply to specific operation modes, can we specify which they work for in the help text?

I added some text for that.
And I also tried to add some details for usage of modes like this🤔

yomek33 · 2024-07-17T10:31:02Z

Additionally, I made the following changes:

Added a license to the project. This also resolved the CI issues. I set the license year to 2022 to match the other files, but please let me know if it should be updated to 2024.
Modified the logic for each Mode. Now, the SeriesCount is adjusted according to the Mode, and the correct SeriesCount is used where necessary.

Please review the updated PR and let me know if you have any further suggestions or concerns.
Thank you!

cstyan

we could have another PR to move the new operation modes to a separate sub command: https://pkg.go.dev/github.com/alecthomas/kingpin#readme-sub-commands

a few more comments about organization/structure of the code

go.mod

cmd/avalanche.go

metrics/serve.go

cstyan · 2024-07-19T03:22:25Z

If you rebase or merge in master it looks like Ben updated the build dependencies recently, so we don't need the change to upgrade to go1.21 anymore. Also, for the DCO check to pass (here) you need to sign commits with git commit -s.

yomek33 · 2024-07-19T13:56:57Z

marged and I signed

metrics/serve.go

Signed-off-by: yomek33 <[email protected]>

… logic Signed-off-by: yomek33 <[email protected]>

Signed-off-by: yomek33 <[email protected]>

∙ fix InvalidSeriesCount logic, add test Signed-off-by: yomek33 <[email protected]>

Signed-off-by: yomek33 <[email protected]>

cstyan

I left a few comments for code change suggestions; these simplify the code a bit and also make sure that the two new operation modes have metrics available for scraping immediately. We either need this change or to set currentSeriesCount = seriesCount in each of the two new modes as you've done in the default case.

cmd/avalanche.go

metrics/serve.go

Signed-off-by: yomek33 <[email protected]>

metrics/serve.go

Signed-off-by: yomek33 <[email protected]>

cstyan · 2024-07-24T18:11:33Z

@yomek33 good work!

yomek33 force-pushed the dynamic-series-adjustment branch from 7e5e09e to 519f6be Compare July 15, 2024 22:07

cstyan reviewed Jul 16, 2024

View reviewed changes

yomek33 force-pushed the dynamic-series-adjustment branch 3 times, most recently from 8acfd37 to aaf4804 Compare July 17, 2024 09:59

yomek33 marked this pull request as ready for review July 17, 2024 10:01

cstyan reviewed Jul 17, 2024

View reviewed changes

go.mod Outdated Show resolved Hide resolved

cmd/avalanche.go Outdated Show resolved Hide resolved

cmd/avalanche.go Outdated Show resolved Hide resolved

metrics/serve.go Show resolved Hide resolved

metrics/serve.go Show resolved Hide resolved

yomek33 force-pushed the dynamic-series-adjustment branch from 1c44292 to 28678fb Compare July 18, 2024 13:54

yomek33 force-pushed the dynamic-series-adjustment branch from 3cc0f5b to 55c0c4a Compare July 19, 2024 13:11

cstyan reviewed Jul 19, 2024

View reviewed changes

metrics/serve.go Outdated Show resolved Hide resolved

yomek33 force-pushed the dynamic-series-adjustment branch from 692db74 to 8d7c0c5 Compare July 19, 2024 22:51

yomek33 added 10 commits July 20, 2024 08:01

Add series doubling and halving functionality

7be3a79

Signed-off-by: yomek33 <[email protected]>

Add series gradual increase and decrease functionlity

79901fa

Signed-off-by: yomek33 <[email protected]>

change seriesOperationMode name, fix test code, change gradual change…

e52cbdd

… logic Signed-off-by: yomek33 <[email protected]>

Standardize and fix seriesCount variable usage in RunMetrics function

4cf527b

Signed-off-by: yomek33 <[email protected]>

add details in help

2ad7cf9

∙ fix InvalidSeriesCount logic, add test Signed-off-by: yomek33 <[email protected]>

Add license text

2d5ae8c

Signed-off-by: yomek33 <[email protected]>

fix go version to 1.21, fix flag help text

cb21a84

Signed-off-by: yomek33 <[email protected]>

go mod tidy

2822c9a

Signed-off-by: yomek33 <[email protected]>

separate logics into its own function

5569a40

Signed-off-by: yomek33 <[email protected]>

Update metrics handling logic

fddb408

Signed-off-by: yomek33 <[email protected]>

yomek33 force-pushed the dynamic-series-adjustment branch from 8d7c0c5 to fddb408 Compare July 19, 2024 23:04

yomek33 added 2 commits July 20, 2024 08:13

go mod tidy

c85b907

Signed-off-by: yomek33 <[email protected]>

gofumpt

8173c0c

Signed-off-by: yomek33 <[email protected]>

cstyan reviewed Jul 22, 2024

View reviewed changes

cmd/avalanche.go Outdated Show resolved Hide resolved

metrics/serve.go Outdated Show resolved Hide resolved

metrics/serve.go Outdated Show resolved Hide resolved

metrics/serve.go Outdated Show resolved Hide resolved

metrics/serve.go Outdated Show resolved Hide resolved

yomek33 added 2 commits July 24, 2024 01:10

copyright 2022->2024, change flug default

7a2f8dd

Signed-off-by: yomek33 <[email protected]>

change currentSeriesCount to be a pointer

bddbb3b

Signed-off-by: yomek33 <[email protected]>

yomek33 force-pushed the dynamic-series-adjustment branch from 77cb088 to bddbb3b Compare July 23, 2024 19:08

cstyan reviewed Jul 23, 2024

View reviewed changes

metrics/serve.go Outdated Show resolved Hide resolved

fix

a41aecf

Signed-off-by: yomek33 <[email protected]>

yomek33 force-pushed the dynamic-series-adjustment branch from 56328e0 to a41aecf Compare July 24, 2024 16:48

cstyan merged commit 9caa882 into prometheus-community:main Jul 24, 2024
6 checks passed

cstyan mentioned this pull request Jul 24, 2024

Some style changes + logging and a changelog entry for new operation modes #69

Merged

yomek33 mentioned this pull request Jul 30, 2024

spike mode for dynamic series adjustment #70

Merged

yomek33 mentioned this pull request Aug 26, 2024

Refactor RunMetrics into Separate Functions for Each Run Mode #73

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamic series adjustment #64

Dynamic series adjustment #64

yomek33 commented Jul 15, 2024 •

edited

Loading

cstyan left a comment

cstyan Jul 16, 2024

yomek33 Jul 17, 2024

cstyan Jul 17, 2024

cstyan Jul 16, 2024

yomek33 Jul 17, 2024

cstyan Jul 16, 2024

yomek33 Jul 17, 2024

cstyan Jul 16, 2024

yomek33 Jul 17, 2024

yomek33 commented Jul 17, 2024 •

edited

Loading

cstyan left a comment

cstyan commented Jul 19, 2024

yomek33 commented Jul 19, 2024

cstyan left a comment

cstyan commented Jul 24, 2024

Dynamic series adjustment #64

Dynamic series adjustment #64

Conversation

yomek33 commented Jul 15, 2024 • edited Loading

double-halve

gradual-change

Tests

cstyan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yomek33 commented Jul 17, 2024 • edited Loading

cstyan left a comment

Choose a reason for hiding this comment

cstyan commented Jul 19, 2024

yomek33 commented Jul 19, 2024

cstyan left a comment

Choose a reason for hiding this comment

cstyan commented Jul 24, 2024

yomek33 commented Jul 15, 2024 •

edited

Loading

yomek33 commented Jul 17, 2024 •

edited

Loading