Support fetching multiple modules in one scrape #945
Conversation
|
Thanks for review. |
|
I think supporting both multiple |
|
Thank you for confirming some points. |
|
Pro tip, |
|
Question: do we actually want the multiple module scrapes to be in parallel? It ought to work of course, but it makes me a bit uncomfortable as I worry about tickling bugs in vendor implementations. Taking this to an extreme, why doesn't snmp_exporter do all its snmpwalks and snmpgets in parallel for a single module (or does it in fact do this?) |
Probably not most of the time. But the default concurrency of 1 is fine. If users know what they're doing, they can do more. Really, we'll eventually want this to be a config option. I'm going to say this ahead of time, not a scrape parameter. That's very dangerous.
I think this has been requested previously. I think if we implemented a module walk de-duplication we would maybe implement it at that time. For now, the current behavior of serial SNMP ops is fine. |
|
If the changes look OK to some extent, I will rebase but also put the SIGNOFF in all commits. |
|
Describe the log of the test. generator.yml http request snmp-exporter log |
|
@candlerb How do you like this compared to your version? |
|
It's a bigger change - for example it creates per-module scrape stats - but if that's OK with you then it's OK with me :-) I can't see any circumstance in which I'd use concurrency>1 (as it can't be controlled on a target-by-target basis), but it doesn't add much complexity to the code, so it's fine as-is. |
|
I would have been previously more objecting to the per-module stats. But with the new separated auths, the number of modules will likely be greatly reduced in real-world installs. |
Signed-off-by: Kakuya Ando <[email protected]>
Signed-off-by: Kakuya Ando <[email protected]>
Signed-off-by: Kakuya Ando <[email protected]>
Signed-off-by: Kakuya Ando <[email protected]>
Signed-off-by: Kakuya Ando <[email protected]>
Signed-off-by: Kakuya Ando <[email protected]>
Signed-off-by: Kakuya Ando <[email protected]>
collector/collector.go
Outdated
| }() | ||
| } | ||
|
|
||
| go func() { |
There was a problem hiding this comment.
Nit: I don't think we need this goroutine, it does not give us anything. We could use "Collect(" (this method) scope to distribute the work and close channel afterwards and then wait. Only readability/complexity nit though.
There was a problem hiding this comment.
Indeed, the last Go routine seems unnecessary. Could you please confirm if my understanding is correct?
--- a/collector/collector.go
+++ b/collector/collector.go
@@ -527,12 +527,10 @@ func (c Collector) Collect(ch chan<- prometheus.Metric) {
}()
}
- go func() {
- for _, module := range c.modules {
- workerChan <- module
- }
- close(workerChan)
- }()
+ for _, module := range c.modules {
+ workerChan <- module
+ }
+ close(workerChan)
wg.Wait()
There was a problem hiding this comment.
Correct. Do you mind writing quick unit test for it?
There was a problem hiding this comment.
Thanks.
I think that unit test for Collector is difficult.
Because Collector does not have an SNMP Fetcher interface, but calls GoSNMP directly in ScrapeTargets.
Please let me know if I am not aware of this and if there is another way to make this easier.
There was a problem hiding this comment.
I agree, writing a test for this is a bit difficult. I'm OK without an additional test here.
| prometheus.GaugeValue, | ||
| time.Since(start).Seconds()) | ||
| } | ||
|
|
||
| // Collect implements Prometheus.Collector. | ||
| func (c Collector) Collect(ch chan<- prometheus.Metric) { |
There was a problem hiding this comment.
I assume there is a unit test for this somewhere?
Signed-off-by: Kakuya Ando <[email protected]>
SuperQ
changed the title
* [CHANGE] Sanitize invalid UTF-8 #968 * [FEATURE] Support fetching multiple modules in one scrape #945 * [FEATURE] Support loading multiple configuration files #970 Signed-off-by: SuperQ <[email protected]>
* [CHANGE] Sanitize invalid UTF-8 #968 * [FEATURE] Support fetching multiple modules in one scrape #945 * [FEATURE] Support loading multiple configuration files #970 Signed-off-by: SuperQ <[email protected]>
* Implemented a feature to fetch from multiple modules * Changed snmp_collection_duration_seconds from Summary to Histogram --------- Signed-off-by: Kakuya Ando <[email protected]> Signed-off-by: Stephan Windischmann <[email protected]>
* [CHANGE] Sanitize invalid UTF-8 prometheus#968 * [FEATURE] Support fetching multiple modules in one scrape prometheus#945 * [FEATURE] Support loading multiple configuration files prometheus#970 Signed-off-by: SuperQ <[email protected]> Signed-off-by: Stephan Windischmann <[email protected]>
I implemented the ability to scrape in multiple modules.
Parallel processing depends on user requirements, so I made it possible to specify the number of parallels. #731
test
generator.yml
output
Confirmation of Significant Changes
Since the change will add module information to the metrics output by snmp_exporter as standard, could you please confirm that this is the right direction for the change?