Add metric for every step on the pipeline

[kaisbaccour]

• beacons_request_inclsuion_in_customer_chain_time
• relay_and_job_subscription_time
• drand_wait_time
• verification_time
• relay_and_block_inclsuion_in_customer_chain_time

[webmaster128]

This means many different histograms, right? So far we have just two, the e2e time and the processing time (i.e. the part we can control).

[kaisbaccour]

This means many different histograms, right? So far we have just two, the e2e time and the processing time (i.e. the part we can control).

Yes indeed so probably 5 histograms. Then the timing out task would get the max_integer.
We could eventually add the ics20 transfer time but that's probably not as important.
For the gateway step we need to make sure that the job enqueue step can happen before the drand release.

Next to this we need to make a list of what visualisations/dashboards we want to see on grafana.

[webmaster128]

When we track all of this data, we need to ensure to not stack the histograms. Some of them track overlapping times such as relaying to the gateway and waiting for the drand.