Hooks for unified operation analysis

[fillest]

The problem: our Aerospike workload has become quite complex and unpredictable, so we need some advanced detailed metrics for analysis of performance problems (which do occur, sometimes with disastrous effect). For example:

• operation rates per set
• read record size statistics per set (network bandwidth analysis)
• something for continuous hot key analysis

The codebase is not small too, so it's quite inconvenient and fragile to manually instrument all the places where Aerospike is involved. So it would be great to have some sort of a hook inside the client where I can access the key, the record size and so on.
Is it possible to add this feature without breaking the api compatibility?
If not, probably it would still be easier for us to make an internal fork. Could you please than suggest which places in the client code should I look at?

[khaf]

This is a bit of an open question for us, because gathering these stats require lots of branching and synchronization on the client-side per transaction, so I'm not sure if that level of wiring would be beneficial without incurring massive performance penalties on the client side, even without activating it.

At the same time, I'm not against the idea, but I need some time to think about it. If you are pressed for a solution right now, the best place to do it would be the client.go, where most API are defined.

We are open for ideas regarding instrumentation in the client from the community.

[un000]

It would be nice to place some kind of an interceptor to handle metrics manually. Maybe this method could live inside the Policy.
Something like:

// RPCStats is a statistics for a completed request.
type RPCStats struct {
	Operation string
	Node      string
	Namespace string
	Key       string
	Set       string

	Status           string
	Duration         time.Duration
	SentBytes        int
	ReceivedBytes    int
	IsWriteOperation bool
}

client.SetStatsHandler(func(stats RPCStats)) {
	my_metrics.Timings.WithLabels(stats.Operation).Add(stats.Duration)
	my_metrics.Requests.WithLabels(stats.Operation, stats.Status).Inc()
	my_metrics.SentBytes.WithLabels(stats.Operation).Add(stats.SentBytes)
	my_metrics.ReceivedBytes.WithLabels(stats.Operation).Add(stats.ReceivedBytes)
}) 

In this case, I think performance degradation won't be so crucial. So every user can implement personal aggregation for his SLA etc. API won't break.

For now, we use own decorators over the client, but the client can expose more metrics to us via standard way.