Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prometheus can monitor peers that are added to the network after startup #2

Open
williambanfield opened this issue Jun 9, 2022 · 1 comment · May be fixed by #7
Open

Prometheus can monitor peers that are added to the network after startup #2

williambanfield opened this issue Jun 9, 2022 · 1 comment · May be fixed by #7
Assignees
@thanethomson

Comments

@williambanfield
Copy link
Contributor

Currently, the prometheus configuration is generated and loaded when the network is first created. To enable nodes to be added to the network, we will need to allow the prometheus monitoring application to receive metrics from new nodes.

Possible options for this include:

  1. Using prometheus in a push based way so that new nodes push metrics to our monitor
  2. Recreating the prometheus configuration when each node is instantiated.
@williambanfield williambanfield changed the title Prometheus can monitor peers that are added to the network late Prometheus can monitor peers that are added to the network after startup Jun 9, 2022
@thanethomson
Copy link
Contributor

thanethomson commented Jun 10, 2022 •
edited
Loading

I see now that the way we did this in GoT was to deploy Telegraf instances on all of the validators to collect all metrics and push them directly to a central InfluxDB instance, to which we connected our Grafana dashboard for querying/monitoring.

@thanethomson thanethomson linked a pull request Jun 17, 2022 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thanethomson thanethomson

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Refactor scripts to facilitate dynamic monitoring interchainio/tendermint-testnet
2 participants
@thanethomson @williambanfield