Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added mtypes CLI for generating realistic avalanche metric type distributions. #97

Merged
merged 1 commit into from
Oct 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
avalanche
./avalanche
.build/
.idea/
4 changes: 3 additions & 1 deletion .promu.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@ repository:
build:
binaries:
- name: avalanche
path: ./cmd
path: ./cmd/avalanche
- name: mtypes
path: ./cmd/mtypes
ldflags: |
-X github.com/prometheus/common/version.Version={{.Version}}
-X github.com/prometheus/common/version.Revision={{.Revision}}
Expand Down
1 change: 1 addition & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ LABEL maintainer="The Prometheus Authors <[email protected]
ARG ARCH="amd64"
ARG OS="linux"
COPY .build/${OS}-${ARCH}/avalanche /bin/avalanche
COPY .build/${OS}-${ARCH}/mtypes /bin/mtypes

EXPOSE 9101
USER nobody
Expand Down
38 changes: 25 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,29 +9,41 @@ This allows load testing services that can scrape (e.g. Prometheus, OpenTelemetr

Metric names and unique series change over time to simulate series churn.

Checkout the [blog post](https://blog.freshtracks.io/load-testing-prometheus-metric-ingestion-5b878711711c).
Checkout the (old-ish) [blog post](https://blog.freshtracks.io/load-testing-prometheus-metric-ingestion-5b878711711c).

## configuration flags
## Installing

### Locally

```bash
avalanche --help
go install github.com/prometheus-community/avalanche/cmd/avalanche@latest
${GOPATH}/bin/avalanche --help
```

## run Docker image
### Docker

```bash
docker run quay.io/prometheuscommunity/avalanche:main --help
docker run quay.io/prometheuscommunity/avalanche:latest --help
```

## Endpoints
NOTE: We recommend using pinned image to a certain version (see all tags [here](https://quay.io/repository/prometheuscommunity/avalanche?tab=tags&tag=latest))

## Using

See [example](example/kubernetes-deployment.yaml) k8s manifest for deploying avalanche as an always running scrape target.

### Configuration

See `--help` for all flags and their documentation.

Notably, from 0.6.0 version, `avalanche` allows specifying various counts per various metric types.

You can choose you own distribution, but usually it makes more sense to mimic realistic distribution used by your example targets. Feel free to use a [handy `mtypes` Go CLI](./cmd/mtypes) to gather type distributions from a target and generate avalanche flags from it.

On top of scrape target functionality, avalanche is capable of Remote Write client load simulation, following the same, configured metric distribution via `--remote*` flags.

### Endpoints

Two endpoints are available :
* `/metrics` - metrics endpoint
* `/health` - healthcheck endpoint

## build and run go binary

```bash
go install github.com/prometheus-community/avalanche/cmd@latest
go/bin/cmd --help
```
File renamed without changes.
53 changes: 53 additions & 0 deletions cmd/mtypes/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# mtypes

Go CLI gathering statistics around the distribution of types, average number of buckets (and more) across your Prometheus metrics/series.

## Usage

The main usage allows to take resource (from stdin, file or HTTP /metrics endpoint) and calculate type statistics e.g.:

```bash
go install github.com/prometheus-community/avalanche/cmd/mtypes@latest # or locally: alias mtypes="go run ./cmd/mtypes"
$ mtypes -resource=http://localhost:9090/metrics
$ mtypes -resource=./cmd/mtypes/exampleprometheustarget.txt
$ cat ./cmd/mtypes/exampleprometheustarget.txt | mtypes
```

```bash
Metric Type Metric Families Series (adjusted) Series (adjusted) % Average Buckets/Objectives
GAUGE 76 93 (93) 31.958763 (16.909091) -
COUNTER 96 157 (157) 53.951890 (28.545455) -
HISTOGRAM 8 14 (186) 4.810997 (33.818182) 11.285714
SUMMARY 15 27 (114) 9.278351 (20.727273) 2.222222
--- --- --- --- ---
* 195 291 (550) 100.000000 (100.000000) -
```

> NOTE: "Adjusted" series, means actual number of individual series stored in Prometheus. Classic histograms and summaries are stored as a set of counters. This is relevant as the cost of indexing new series is higher than storing complex values (this is why we slowly move to native histograms).

Additionally, you can pass `--avalanche-flags-for-adjusted-series=10000` to print Avalanche v0.6.0+ flags to configure, for avalanche to generate metric target with the given amount of adjusted series, while maintaining a similar distribution e.g.

```bash
cat ./cmd/mtypes/exampleprometheustarget.txt | mtypes --avalanche-flags-for-adjusted-series=1000
Metric Type Metric Families Series (adjusted) Series (adjusted) % Average Buckets/Objectives
GAUGE 76 93 (93) 31.958763 (16.909091) -
COUNTER 96 157 (157) 53.951890 (28.545455) -
HISTOGRAM 8 14 (186) 4.810997 (33.818182) 11.285714
SUMMARY 15 27 (114) 9.278351 (20.727273) 2.222222
--- --- --- --- ---
* 195 291 (550) 100.000000 (100.000000) -

Avalanche flags for the similar distribution to get to the adjusted series goal of: 1000
--gauge-metric-count=16
--counter-metric-count=28
--histogram-metric-count=2
--histogram-metric-bucket-count=10
--native-histogram-metric-count=0
--summary-metric-count=5
--summary-metric-objective-count=2
--series-count=10
--value-interval=300 # Changes values every 5m.
--series-interval=3600 # 1h series churn.
--metric-interval=0
bwplotka marked this conversation as resolved.
Show resolved Hide resolved
This should give the total adjusted series to: 900
```
Loading
Loading