issue-465 Create a documentation section to use Grafana DataSource with SonataFlow Prometheus metrics #693

jianrongzhang89 · 2024-12-10T09:30:38Z

Fix apache/incubator-kie-kogito-serverless-operator#465

Update the document to include Prometheus and Grafana installation, and Grafana Data Source congfiguration and import the default dashboard.

You have read the contributions doc
Pull Request title is properly formatted: Issue-XYZ Subject
Pull Request title contains the target branch if not targeting main: [0.9.x] Issue-XYZ Subject
The nav.adoc file has a link to this guide in the proper category
The index.adoc file has a card to this guide in the proper category, with a meaningful description

ricardozanini · 2024-12-10T22:56:33Z

@jianrongzhang89 can you please take a look on CI?

github-actions · 2024-12-11T10:23:04Z

🎊 PR Preview ea1866d has been successfully built and deployed. See the documentation preview: https://sonataflow-docs-preview-pr-693.surge.sh

jianrongzhang89 · 2024-12-11T10:23:07Z

@jianrongzhang89 can you please take a look on CI?

@ricardozanini fixed CI errors.

…ith SonataFlow Prometheus metrics

wmedvede

I have followed all the document for the OpenShift installation and worked fine.
See image below with my workflows.

Guide is working.
LGTM

wmedvede · 2024-12-11T14:06:25Z

serverlessworkflow/modules/ROOT/pages/cloud/operator/monitoring-workflows.adoc

+image::cloud/operator/monitoring/grafana-dashboard-example.png[]
+
+=== Customize or build your own dashboard
+You can customize or build your own dashboard. For more information, see xref:https://grafana.com/docs/grafana/latest/dashboards[Grafana Dashboards] and xref:cloud/operator/sonataflow-metrics.adoc[SonataFlow Metrics].


This link: xref:https://grafana.com/docs/grafana/latest/dashboards[Grafana Dashboards] is not working.
I think that for external links you must use the link: tag instead.

wmedvede · 2024-12-11T14:06:43Z

serverlessworkflow/modules/ROOT/pages/cloud/operator/monitoring-workflows.adoc

+== Additional resources
+
+* xref:cloud/operator/sonataflow-metrics.adoc[SonataFlow Metrics]
+* xref:https://grafana.com/docs/grafana/latest/dashboards[Grafana Dashboards]


same here, non working link.

wmedvede · 2024-12-11T14:21:49Z

serverlessworkflow/modules/ROOT/pages/cloud/operator/monitoring-workflows.adoc

+    secureJsonData:
+        httpHeaderValue1: 'Bearer ${TOKEN}'
+    name: Prometheus
+    url: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091


So, access to metrics is in the end "protected", and can be accessed only if we give the cluster-monitoring-view to grafana, right?

That's correct: the metrics are exposed to Prometheus but are not available in Grafana.

wmedvede · 2024-12-11T14:35:36Z

serverlessworkflow/modules/ROOT/pages/cloud/operator/sonataflow-metrics.adoc

@@ -0,0 +1,99 @@
+= SonataFlow Metrics


I think this page is not shown in the menu Cloud -> Operator or in any other menu entry.

On the other hand, we have a sort of metrics page that shows the metrics and that I personally don't like 100%.
I think that what we must do, is to do a kind of merge between this metrics content and what is shown in the page below, and provide something good.
But , it's out of from @jianrongzhang89 scope.

see below:

Feel free to merge as is, to not loose this content and we can restructure in a followup PR.
@ricardozanini @domhanak

@ricardozanini @wmedvede @domhanak I tried to merge them. Please review.

wmedvede · 2024-12-11T14:39:25Z

Would you mind check the procedure for regular Kubernetes clusters? @domhanak

wmedvede · 2024-12-12T12:33:33Z

serverlessworkflow/modules/ROOT/pages/_common-content/sonataflow-metrics.adoc

@@ -0,0 +1,134 @@
+== Overview


@jianrongzhang89 this common sonataflow_metrics document is great, many thanks.
My only observation is that the order of occurrence of each metric in the document, is not the same as the one being shown in the initial paragraph, which somehow corresponds with the workflow "natural" life-cycle.

see:

Good suggestion. Done!

wmedvede · 2024-12-12T12:34:44Z

serverlessworkflow/modules/ROOT/pages/_common-content/sonataflow-metrics.adoc

+
+== Metrics Description
+=== kogito_process_instance_completed_total
+Workflow instances that have reached a terminal status, “Aborted” or “Completed”, and thus are considered as completed.


Suggested change

Workflow instances that have reached a terminal status, “Aborted” or “Completed”, and thus are considered as completed.

Workflow instances that have reached a terminal status, `Aborted` or `Completed`, and thus are considered as completed.

wmedvede · 2024-12-12T12:35:25Z

serverlessworkflow/modules/ROOT/pages/_common-content/sonataflow-metrics.adoc

+
+[NOTE]
+====
+These are the only two terminal status. The “Error” state is not terminal.


Suggested change

These are the only two terminal status. The “Error” state is not terminal.

These are the only two terminal status. The `Error` state is not terminal.

wmedvede · 2024-12-12T12:37:27Z

serverlessworkflow/modules/ROOT/pages/_common-content/sonataflow-metrics.adoc

+
+[NOTE]
+====
+This includes workflow instances that are in the "Error" state, since the error state is not a terminal state.


Suggested change

This includes workflow instances that are in the "Error" state, since the error state is not a terminal state.

This includes workflow instances that are in the `Error` state, since the error state is not a terminal state.

wmedvede · 2024-12-12T12:38:08Z

serverlessworkflow/modules/ROOT/pages/_common-content/sonataflow-metrics.adoc

+[NOTE]
+====
+This includes workflow instances that are in the "Error" state, since the error state is not a terminal state.
+Process instances that have reached a terminal status, i.e. "Completed" or "Aborted", are not present in this metric.


Suggested change

Process instances that have reached a terminal status, i.e. "Completed" or "Aborted", are not present in this metric.

Process instances that have reached a terminal status, i.e. `Completed` or `Aborted`, are not present in this metric.

wmedvede · 2024-12-12T12:38:59Z

serverlessworkflow/modules/ROOT/pages/_common-content/sonataflow-metrics.adoc

+----
+
+=== kogito_process_instance_duration_seconds
+Calculates duration of a workflow instance that has reached a terminal state,, i.e. "Aborted" or "Completed". This metric is registered when the process reaches the terminal state.


Suggested change

Calculates duration of a workflow instance that has reached a terminal state,, i.e. "Aborted" or "Completed". This metric is registered when the process reaches the terminal state.

Calculates duration of a workflow instance that has reached a terminal state, i.e. `Aborted` or `Completed`. This metric is registered when the process reaches the terminal state.

wmedvede · 2024-12-12T12:40:16Z

serverlessworkflow/modules/ROOT/pages/cloud/operator/monitoring-workflows.adoc

+= Monitoring Workflows
+:compat-mode!:
+// Metadata:
+:description: Workflows monitoring configuration configuration


Suggested change

:description: Workflows monitoring configuration configuration

:description: Workflows monitoring configuration

ricardozanini

Many thanks, @jianrongzhang89. This documentation seems good. Thanks, @wmedvede, for verifying the steps in the cluster!

ricardozanini · 2024-12-12T17:50:37Z

@kaldesai mind taking a look too?

wmedvede

Hi @jianrongzhang89 , I couldn't evict adding some more nitpicks when re-reading 😄

wmedvede · 2024-12-13T11:28:28Z

serverlessworkflow/modules/ROOT/pages/_common-content/sonataflow-metrics.adoc

+
+In {product_name}, you can check the following metrics:
+
+* `kogito_process_instance_started_total`: Number of started workflows (a workflow that has started might be running or completed)


Suggested change

* `kogito_process_instance_started_total`: Number of started workflows (a workflow that has started might be running or completed)

* `kogito_process_instance_started_total`: Number of started workflows.

wmedvede · 2024-12-13T11:29:14Z

serverlessworkflow/modules/ROOT/pages/_common-content/sonataflow-metrics.adoc

+
+* `kogito_process_instance_started_total`: Number of started workflows (a workflow that has started might be running or completed)
+* `kogito_process_instance_running_total`: Number of running workflows
+* `kogito_process_instance_completed_total`: Number of completed workflows


Suggested change

* `kogito_process_instance_completed_total`: Number of completed workflows

* `kogito_process_instance_completed_total`: Number of completed workflows.

wmedvede · 2024-12-13T11:29:37Z

serverlessworkflow/modules/ROOT/pages/_common-content/sonataflow-metrics.adoc

+* `kogito_process_instance_started_total`: Number of started workflows (a workflow that has started might be running or completed)
+* `kogito_process_instance_running_total`: Number of running workflows
+* `kogito_process_instance_completed_total`: Number of completed workflows
+* `kogito_process_instance_error`: Number of workflows that report an error ( a workflow with an error might be still running or have been completed) 


Suggested change

* `kogito_process_instance_error`: Number of workflows that report an error ( a workflow with an error might be still running or have been completed)

* `kogito_process_instance_error`: Number of workflows that report an error.

wmedvede · 2024-12-13T11:30:43Z

serverlessworkflow/modules/ROOT/pages/_common-content/sonataflow-metrics.adoc

+* `kogito_process_instance_running_total`: Number of running workflows
+* `kogito_process_instance_completed_total`: Number of completed workflows
+* `kogito_process_instance_error`: Number of workflows that report an error ( a workflow with an error might be still running or have been completed) 
+* `kogito_process_instance_duration_seconds`: Duration of a process instance in seconds


Suggested change

* `kogito_process_instance_duration_seconds`: Duration of a process instance in seconds

* `kogito_process_instance_duration_seconds`: Duration of a workflow instance in seconds.

wmedvede · 2024-12-13T11:31:34Z

serverlessworkflow/modules/ROOT/pages/_common-content/sonataflow-metrics.adoc

+* `kogito_process_instance_completed_total`: Number of completed workflows
+* `kogito_process_instance_error`: Number of workflows that report an error ( a workflow with an error might be still running or have been completed) 
+* `kogito_process_instance_duration_seconds`: Duration of a process instance in seconds
+* `kogito_node_instance_duration_milliseconds`: Duration of relevant nodes in milliseconds (a workflow is composed by nodes, user might be interested on the time consumed by an specific node type) 


Suggested change

* `kogito_node_instance_duration_milliseconds`: Duration of relevant nodes in milliseconds (a workflow is composed by nodes, user might be interested on the time consumed by an specific node type)

* `kogito_node_instance_duration_milliseconds`: Duration of relevant nodes in milliseconds.

wmedvede · 2024-12-13T11:32:51Z

serverlessworkflow/modules/ROOT/pages/_common-content/sonataflow-metrics.adoc

+
+[NOTE]
+====
+Internally, workflows are referred as processes. Therefore, the `processId` and `processName` is workflow ID and name respectively.


Suggested change

Internally, workflows are referred as processes. Therefore, the `processId` and `processName` is workflow ID and name respectively.

Internally, workflows are referred as processes. Therefore, the `processId` and `processName` are workflow id and name respectively.

wmedvede · 2024-12-13T11:33:26Z

serverlessworkflow/modules/ROOT/pages/_common-content/sonataflow-metrics.adoc

+Internally, workflows are referred as processes. Therefore, the `processId` and `processName` is workflow ID and name respectively.
+====
+
+Each of the metrics mentioned previously contains a label for a specific workflow ID. For example, the `kogito_process_instance_completed_total` metric below contains the labels for `callbackstatetimeouts` workflow:


Suggested change

Each of the metrics mentioned previously contains a label for a specific workflow ID. For example, the `kogito_process_instance_completed_total` metric below contains the labels for `callbackstatetimeouts` workflow:

Each of the metrics mentioned previously contains a label for a specific workflow id. For example, the `kogito_process_instance_completed_total` metric below contains the labels for `callbackstatetimeouts` workflow:

wmedvede · 2024-12-13T11:41:14Z

serverlessworkflow/modules/ROOT/pages/_common-content/sonataflow-metrics.adoc

+----
+
+=== kogito_process_instance_duration_seconds
+Calculates duration of a workflow instance that has reached a terminal state,, i.e. `Aborted` or `Completed`. This metric is registered when the process reaches the terminal state.


Suggested change

Calculates duration of a workflow instance that has reached a terminal state,, i.e. `Aborted` or `Completed`. This metric is registered when the process reaches the terminal state.

Calculates duration of a workflow instance that has reached a terminal state, i.e. `Aborted` or `Completed`. This metric is registered when the process reaches the terminal state.

wmedvede · 2024-12-13T11:43:25Z

serverlessworkflow/modules/ROOT/pages/_common-content/sonataflow-metrics.adoc

+* `kogito_process_instance_error`: Number of workflows that report an error ( a workflow with an error might be still running or have been completed) 
+* `kogito_process_instance_duration_seconds`: Duration of a process instance in seconds
+* `kogito_node_instance_duration_milliseconds`: Duration of relevant nodes in milliseconds (a workflow is composed by nodes, user might be interested on the time consumed by an specific node type) 
+* `sonataflow_input_parameters_counter`: Records input parameters, the occurrences of <"param_name","param_value"> per `processId`.


Suggested change

* `sonataflow_input_parameters_counter`: Records input parameters, the occurrences of <"param_name","param_value"> per `processId`.

* `sonataflow_input_parameters_counter_total`: Records input parameters, the occurrences of <"param_name","param_value"> per `processId`.

…ith SonataFlow Prometheus metrics: address review comments

wmedvede

Hi @RichardW98 a few nitpicks, and I have also re-installed the grafana dahsboard after these last modification.
Is working good, great work!

Just an observation regarding the dashboard, see screenshots please:

In my tests, I have these workflows: callbackstatetimeouts and callbackstatetimeouts-gitops.

The dashboard works fine:

However, in the filters a "greeting" value is shown.

serverlessworkflow/modules/ROOT/pages/_common-content/sonataflow-metrics.adoc

serverlessworkflow/modules/ROOT/pages/cloud/operator/monitoring-workflows.adoc

…ow-metrics.adoc Co-authored-by: Walter Medvedeo <[email protected]>

jianrongzhang89 · 2024-12-21T01:48:31Z

@wmedvede I updated PR based on your above comments. Thanks.

…ith SonataFlow Prometheus metrics: address review comments

jianrongzhang89 requested review from domhanak, ricardozanini and krisv as code owners December 10, 2024 09:30

jianrongzhang89 force-pushed the monitoring branch from dd9e4a9 to 1537dee Compare December 10, 2024 09:47

jianrongzhang89 force-pushed the monitoring branch 2 times, most recently from 3250a02 to d95b765 Compare December 11, 2024 10:22

[issue 465]Create a documentation section to use Grafana DataSource w…

79a31d8

…ith SonataFlow Prometheus metrics

jianrongzhang89 force-pushed the monitoring branch from d95b765 to 79a31d8 Compare December 11, 2024 10:32

wmedvede approved these changes Dec 11, 2024

View reviewed changes

jianrongzhang89 force-pushed the monitoring branch 2 times, most recently from eb1bac5 to 4892c25 Compare December 11, 2024 21:10

wmedvede requested changes Dec 12, 2024

View reviewed changes

ricardozanini approved these changes Dec 12, 2024

View reviewed changes

jianrongzhang89 force-pushed the monitoring branch from 4892c25 to c406023 Compare December 12, 2024 19:25

wmedvede requested changes Dec 13, 2024

View reviewed changes

jianrongzhang89 force-pushed the monitoring branch from c406023 to 1a39e31 Compare December 13, 2024 12:05

[issue 465]Create a documentation section to use Grafana DataSource w…

80de021

…ith SonataFlow Prometheus metrics: address review comments

jianrongzhang89 force-pushed the monitoring branch from 1a39e31 to 80de021 Compare December 20, 2024 01:46

wmedvede requested changes Dec 20, 2024

View reviewed changes

jianrongzhang89 and others added 2 commits December 20, 2024 18:22

Update serverlessworkflow/modules/ROOT/pages/_common-content/sonatafl…

dc84901

…ow-metrics.adoc Co-authored-by: Walter Medvedeo <[email protected]>

Update serverlessworkflow/modules/ROOT/pages/_common-content/sonatafl…

4504bac

…ow-metrics.adoc Co-authored-by: Walter Medvedeo <[email protected]>

jianrongzhang89 force-pushed the monitoring branch from d34b6fe to c58e684 Compare December 21, 2024 01:50

[issue 465]Create a documentation section to use Grafana DataSource w…

681b553

…ith SonataFlow Prometheus metrics: address review comments

jianrongzhang89 force-pushed the monitoring branch from c58e684 to 681b553 Compare December 21, 2024 23:14

	Workflow instances that have reached a terminal status, “Aborted” or “Completed”, and thus are considered as completed.
	Workflow instances that have reached a terminal status, `Aborted` or `Completed`, and thus are considered as completed.

	These are the only two terminal status. The “Error” state is not terminal.
	These are the only two terminal status. The `Error` state is not terminal.

	This includes workflow instances that are in the "Error" state, since the error state is not a terminal state.
	This includes workflow instances that are in the `Error` state, since the error state is not a terminal state.

	Process instances that have reached a terminal status, i.e. "Completed" or "Aborted", are not present in this metric.
	Process instances that have reached a terminal status, i.e. `Completed` or `Aborted`, are not present in this metric.

	Calculates duration of a workflow instance that has reached a terminal state,, i.e. "Aborted" or "Completed". This metric is registered when the process reaches the terminal state.
	Calculates duration of a workflow instance that has reached a terminal state, i.e. `Aborted` or `Completed`. This metric is registered when the process reaches the terminal state.

	:description: Workflows monitoring configuration configuration
	:description: Workflows monitoring configuration


		In {product_name}, you can check the following metrics:

		* `kogito_process_instance_started_total`: Number of started workflows (a workflow that has started might be running or completed)

	* `kogito_process_instance_completed_total`: Number of completed workflows
	* `kogito_process_instance_completed_total`: Number of completed workflows.

	* `kogito_process_instance_error`: Number of workflows that report an error ( a workflow with an error might be still running or have been completed)
	* `kogito_process_instance_error`: Number of workflows that report an error.

	* `kogito_process_instance_duration_seconds`: Duration of a process instance in seconds
	* `kogito_process_instance_duration_seconds`: Duration of a workflow instance in seconds.

	* `kogito_node_instance_duration_milliseconds`: Duration of relevant nodes in milliseconds (a workflow is composed by nodes, user might be interested on the time consumed by an specific node type)
	* `kogito_node_instance_duration_milliseconds`: Duration of relevant nodes in milliseconds.

	Internally, workflows are referred as processes. Therefore, the `processId` and `processName` is workflow ID and name respectively.
	Internally, workflows are referred as processes. Therefore, the `processId` and `processName` are workflow id and name respectively.

	Each of the metrics mentioned previously contains a label for a specific workflow ID. For example, the `kogito_process_instance_completed_total` metric below contains the labels for `callbackstatetimeouts` workflow:
	Each of the metrics mentioned previously contains a label for a specific workflow id. For example, the `kogito_process_instance_completed_total` metric below contains the labels for `callbackstatetimeouts` workflow:

	* `sonataflow_input_parameters_counter`: Records input parameters, the occurrences of <"param_name","param_value"> per `processId`.
	* `sonataflow_input_parameters_counter_total`: Records input parameters, the occurrences of <"param_name","param_value"> per `processId`.

issue-465 Create a documentation section to use Grafana DataSource with SonataFlow Prometheus metrics #693

Are you sure you want to change the base?

issue-465 Create a documentation section to use Grafana DataSource with SonataFlow Prometheus metrics #693

Conversation

jianrongzhang89 commented Dec 10, 2024 • edited by ricardozanini Loading

ricardozanini commented Dec 10, 2024

github-actions bot commented Dec 11, 2024 • edited Loading

jianrongzhang89 commented Dec 11, 2024

wmedvede left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wmedvede commented Dec 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ricardozanini left a comment

Choose a reason for hiding this comment

ricardozanini commented Dec 12, 2024

wmedvede left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wmedvede left a comment

Choose a reason for hiding this comment

jianrongzhang89 commented Dec 21, 2024

jianrongzhang89 commented Dec 10, 2024 •

edited by ricardozanini

Loading

github-actions bot commented Dec 11, 2024 •

edited

Loading