Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DOCS-7747, DOCS-7795 Adds APM Deployment Tracking Documentation #22866

Merged
merged 63 commits into from
May 16, 2024
Merged
Show file tree
Hide file tree
Changes from 51 commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
b43f08c
update structure
kassenq Apr 24, 2024
2f1880e
start on apm doc
kassenq Apr 24, 2024
9928acb
draft deployments page
kassenq Apr 25, 2024
02a6519
update setup page
kassenq Apr 25, 2024
35c704a
update apm docs
kassenq Apr 25, 2024
4ec1c23
move setup instructions from DORA home page to setup page
kassenq Apr 25, 2024
f51f03d
clarify metrics wording
kassenq Apr 25, 2024
e591b65
Merge branch 'master' into kassenq/dora-apm-docs
alai97 Apr 25, 2024
21d5886
remove deployment tracking non-github limitation
kassenq Apr 26, 2024
f0461be
Merge Changes
alai97 Apr 26, 2024
27577cb
Update apm.md
kassenq Apr 29, 2024
c05804e
add api page and consolidate CLT to index page
kassenq Apr 29, 2024
3270189
metric explanation
kassenq Apr 29, 2024
798a8ed
Merge branch 'master' into kassenq/dora-apm-docs
kassenq Apr 29, 2024
a7e3c41
Add Pre-filled Query
alai97 Apr 29, 2024
0657b95
update accuracy of setup page
kassenq May 1, 2024
cf3bcf5
remove repetitive setup steps in dora index page
kassenq May 1, 2024
5e6694a
update capitalization
kassenq May 1, 2024
550c278
update nav to account for api pages
kassenq May 1, 2024
2bcaf4c
create incident api page and update deployment api page
kassenq May 1, 2024
9a97c12
move events explorer into data collected page and update page
kassenq May 1, 2024
226b6f9
modify setup index page
kassenq May 1, 2024
4fef130
incident to failure renaming and capitalization
kassenq May 1, 2024
2535bf7
file and nav renaming to failures
kassenq May 1, 2024
067fbcb
add knowledge center article
kassenq May 2, 2024
2a6ab94
Doc Review
alai97 May 2, 2024
e5efc51
Adding PagerDuty documentation
andrea-mosk May 6, 2024
f9ba10a
Update content/en/dora_metrics/_index.md
kassenq May 6, 2024
0a2148c
Update content/en/dora_metrics/deployments/_index.md
kassenq May 6, 2024
c0dacc9
update change lead time section in deployment index page
kassenq May 6, 2024
372ca06
Update content/en/dora_metrics/deployments/deployment_api.md
kassenq May 6, 2024
42151a9
remove private beta callouts
kassenq May 6, 2024
5381d74
consolidate metric calculation descriptions to main page
kassenq May 6, 2024
25c06b2
update page headings for consistency
kassenq May 6, 2024
c12e6eb
clarify that incidents are a proxy for failed deployments
kassenq May 6, 2024
7e44406
update further reading links
kassenq May 6, 2024
b5dfe07
SME Review
alai97 May 6, 2024
326f221
Doc Review
alai97 May 6, 2024
376393a
Link Fixes
alai97 May 6, 2024
ca7d20c
Use Markdown Table
alai97 May 6, 2024
dc319d7
Update content/en/dora_metrics/failures/_index.md
kassenq May 7, 2024
53ddc8e
SME Review
alai97 May 7, 2024
120e7aa
One More
alai97 May 7, 2024
823cecb
Add Public Beta Banner
alai97 May 7, 2024
91b7ca5
One More
alai97 May 8, 2024
37dc0e1
commit changes
kassenq May 9, 2024
33c6460
Merge remote-tracking branch 'refs/remotes/origin/kassenq/dora-apm-do…
kassenq May 9, 2024
09264c2
update further reading links
kassenq May 9, 2024
5321ef4
update further reading and shorten API options
kassenq May 14, 2024
a4c2d68
add images
kassenq May 14, 2024
f6d98d5
higher resolution images
kassenq May 14, 2024
db70681
Update content/en/dora_metrics/data_collected/_index.md
kassenq May 16, 2024
305e21b
Update content/en/dora_metrics/failures/_index.md
kassenq May 16, 2024
2e8ecb3
Update content/en/dora_metrics/failures/incident_api.md
kassenq May 16, 2024
909e4e4
Update content/en/dora_metrics/deployments/_index.md
kassenq May 16, 2024
bb126b3
Update content/en/dora_metrics/deployments/deployment_api.md
kassenq May 16, 2024
f2905d2
Update content/en/dora_metrics/failures/_index.md
kassenq May 16, 2024
6b9325a
Update content/en/dora_metrics/_index.md
kassenq May 16, 2024
01806f8
Update content/en/dora_metrics/deployments/_index.md
kassenq May 16, 2024
814b0b1
Update content/en/dora_metrics/deployments/apm.md
kassenq May 16, 2024
35c904d
Merge branch 'master' into kassenq/dora-apm-docs
kassenq May 16, 2024
af37f7d
update links
rtrieu May 16, 2024
a2f5e45
Merge branch 'master' into kassenq/dora-apm-docs
kassenq May 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 35 additions & 10 deletions config/_default/menus/main.en.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3729,16 +3729,41 @@ menu:
parent: dora_metrics
identifier: dora_metrics_setup
weight: 1
- name: Deployment events
url: dora_metrics/setup/deployments
parent: dora_metrics_setup
identifier: dora_metrics_setup_deployments
weight: 101
- name: Incident events
url: dora_metrics/setup/incidents
parent: dora_metrics_setup
identifier: dora_metrics_setup_incidents
weight: 102
- name: Deployment Data Sources
url: dora_metrics/deployments
parent: dora_metrics
identifier: dora_metrics_deployments
weight: 2
- name: APM Deployment Tracking
url: dora_metrics/deployments/apm
parent: dora_metrics_deployments
identifier: dora_metrics_deployment_apm
weight: 201
- name: API
url: dora_metrics/deployments/deployment_api
parent: dora_metrics_deployments
identifier: dora_metrics_deployment_events_api
weight: 202
- name: Failure Data Sources
url: dora_metrics/failures/
parent: dora_metrics
identifier: dora_metrics_incidents
weight: 3
- name: PagerDuty
url: dora_metrics/failures/pagerduty
parent: dora_metrics_incidents
identifier: dora_metrics_incident_pagerduty
weight: 301
- name: API
url: dora_metrics/failures/incident_api
parent: dora_metrics_incidents
identifier: dora_metrics_incident_events_api
weight: 302
- name: Data Collected
url: dora_metrics/data_collected/
parent: dora_metrics
identifier: dora_metrics_data_collected
weight: 4
- name: Observability Pipelines
url: observability_pipelines/
pre: pipelines
Expand Down
87 changes: 24 additions & 63 deletions content/en/dora_metrics/_index.md
Original file line number Diff line number Diff line change
@@ -1,111 +1,72 @@
---
title: DORA Metrics
kind: documentation
description: Learn how to use DORA metrics to measure and improve software development.
description: Learn how to use DORA metrics to measure and improve your organization's software delivery processes.
aliases:
- /continuous_integration/dora_metrics
is_beta: true
further_reading:
- link: "https://app.datadoghq.com/release-notes?category=Software%20Delivery"
tag: "Release Notes"
text: "Check out the latest Software Delivery releases! (App login required)"
- link: "/continuous_integration/pipelines"
tag: "Documentation"
text: "Learn about Pipeline Visibility"
- link: "/continuous_delivery/deployments"
tag: "Documentation"
text: "Learn about Deployment Visibility"
- link: "/continuous_integration/tests"
- link: "/service_management/events"
tag: "Documentation"
text: "Learn about Test Visibility"
- link: "/code_analysis"
text: "Learn about Event Management"
- link: "/monitors/types/metric"
tag: "Documentation"
text: "Learn about Code Analysis"
text: "Learn about Metric Monitors"
---

{{< site-region region="gov" >}}
<div class="alert alert-warning">DORA Metrics is not available in the selected site ({{< region-param key="dd_site_name" >}}) at this time.</div>
{{< /site-region >}}

{{< callout url="https://forms.gle/Eqq6uXfGjYxmqpjDA" header="false" >}}
The DORA Metrics private beta is closed. Fill out the form below to be added to the waitlist.
{{< callout url="#" btn_hidden="true" header="Try the Beta!" >}}
DORA Metrics is in public beta.
{{< /callout >}}

## Overview

DevOps Research and Assessment (DORA) metrics are [four key metrics][1] used to indicate the velocity and stability of software development.
DevOps Research and Assessment (DORA) metrics are [four key metrics][1] that indicate the velocity and stability of software development.

Deployment Frequency
Deployment frequency
: How often an organization successfully releases to production.

Lead Time for Changes
Lead time for changes
: The amount of time it takes a commit to get into production.

Change Failure Rate
Change failure rate
: The percentage of deployments causing a failure in production.

Time to Restore Service
Time to restore service
: How long it takes an organization to recover from a failure in production.

Defining and tracking DORA metrics can help you identify areas of improvement for your team or organization's speed and quality of software delivery.

## Set up DORA Metrics

The four DORA Metrics are calculated based on two types of events:
- **Deployment events**: Indicate that a new deployment has occurred for a service in a specific environment.
Deployment events are used to compute Deployment Frequency, Change Lead Time, and Change Failure Rate.
- **Incident events**: Indicate that a new issue has occurred for a service in a specific environment.
Incident events are used to compute Change Failure Rate and Mean Time to Restore.

{{< whatsnext desc="Set up DORA Metrics in Datadog:" >}}
{{< nextlink href="continuous_integration/dora_metrics/setup/deployments" >}}Send Deployment Events{{< /nextlink >}}
{{< nextlink href="continuous_integration/dora_metrics/setup/incidents" >}}Send Incident Events{{< /nextlink >}}
{{< /whatsnext >}}

## Use DORA Metrics

You can access and visualize your DORA metrics and filter them by team, service, repository, environment, and time period on the [**DORA Metrics** page][2].

Use the information on this page to identify improvements or regressions for each metric, visualize changes, and compare trends over time. DORA metrics can be exported to dashboards or notebooks and be alerted on using [metric monitors][3].

The metrics can also be queried with the [Query timeseries points][4] and [Query timeseries data across multiple products][5] API endpoints.

The metrics provided by DORA Metrics are:

| Metric | Type | Description |
| :--- | :--- | :--- |
| `dora.deployments.count` | count | Used for Deployment Frequency.
| `dora.change_lead_time` | distribution | Contains the age in `seconds` of the git commits at the time of deployment.
| `dora.incidents_impact` | count | Tracks the services or teams impacted by incidents. Used for Change Failure Rate with the formula `dora.incidents_impact / dora.deployments.count`. A big time rollup of at least 1 week is recommended to account for time difference between deployments and when the impact starts.
| `dora.time_to_restore` | distribution | Contains the time in `seconds` between the incident's `started_at` and `finished_at`.

All the metrics contain the following tags when available:
- `service`
- `team`
- `env`
- `repository_id`

**Note**: The `severity` tag is available for the `dora.incidents_impact` and `dora.time_to_restore` metrics, if provided through the API.
To start configuring data sources to send deployment and incident events to Datadog, see the [Setup documentation][2].

### Deployment and incident events
## Analyze DORA Metrics

DORA Metrics also provides individual `deployment`, `incident`, and `incident_finished` events in [Event Management][6] with `source:software_delivery_insights`.
Once you've set up the data sources for your deployment and failure events, navigate to [**Software Delivery** > **DORA Metrics**][4] to identify improvements or regressions for each metric, aggregate them by service or environment, and compare trends over time.

The events can be queried and visualized with the [Events Explorer][7].
{{< img src="dora_metrics/dora_page_light.png" alt="An Overview of your DORA Metrics calculations and insights in the last week" style="width:100%;" >}}
kassenq marked this conversation as resolved.
Show resolved Hide resolved

### Limitations
You can access and visualize your DORA metrics and filter them by team, service, repository, environment, and time period on the [**DORA Metrics** page][4]. For more information about the metrics calculated by DORA Metrics, see the [Data Collected documentation][3].

- Deployment and incident events must be sent as soon as possible. Events for which the `finished_at` timestamp is 1 hour older than the current time are not accepted.
- Deployments or incidents of the same service cannot occur at the same second.
You can also export your DORA Metrics data to dashboards or notebooks and create [metric monitors][5] to trigger alerts on your metrics. For more information, see the [Monitors documentation][6].

## Further Reading

{{< partial name="whats-next/whats-next.html" >}}

[1]: https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance
[2]: https://app.datadoghq.com/ci/dora
[3]: https://docs.datadoghq.com/monitors/types/metric/?tab=threshold
[4]: https://docs.datadoghq.com/api/latest/metrics/#query-timeseries-points
[5]: https://docs.datadoghq.com/api/latest/metrics/#query-timeseries-data-across-multiple-products
[6]: https://app.datadoghq.com/event/explorer?query=source%3Asoftware_delivery_insights
[7]: https://docs.datadoghq.com/service_management/events/explorer/
[1]: https://www.datadoghq.com/knowledge-center/dora-metrics/
[2]: /dora_metrics/setup/
[3]: /dora_metrics/data_collected/
[4]: https://app.datadoghq.com/ci/dora
[5]: /monitors/types/metric/?tab=threshold
[6]: /monitors/
92 changes: 92 additions & 0 deletions content/en/dora_metrics/data_collected/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
---
title: DORA Metrics Data Collected
kind: documentation
further_reading:
- link: '/dora_metrics/'
tag: 'Documentation'
text: 'Learn about DORA Metrics'
- link: '/dora_metrics/setup/'
tag: 'Documentation'
text: 'Set up data sources for DORA Metrics'
- link: '/metrics/'
tag: 'Documentation'
text: 'Learn about metrics'
- link: '/getting_started/tagging/'
tag: 'Documentation'
text: 'Getting Started with Tags'
kassenq marked this conversation as resolved.
Show resolved Hide resolved
---

{{< site-region region="gov" >}}
<div class="alert alert-warning">DORA Metrics is not available in the selected site ({{< region-param key="dd_site_name" >}}) at this time.</div>
{{< /site-region >}}

{{< callout url="#" btn_hidden="true" header="Try the Beta!" >}}
DORA Metrics is in public beta.
{{< /callout >}}

## Overview

DORA Metrics generates [metrics][9] for each one of the four core DORA Metrics, as well as events with associated tags and attributes that are available in the [Events Explorer][1].

## Default metrics

DORA Metrics provides the following default metrics:

| Metric | Type | Description |
| :--- | :--- | :--- |
| `dora.deployments.count` | count | The number of deployments detected by Datadog based on your selected [deployment data source][10].
| `dora.change_lead_time` | distribution | The age in `seconds` of associated Git commits at the time of deployment.
| `dora.incidents_impact` | count | Tracks the services or teams impacted by incidents. Used for change failure rate with the formula `dora.incidents_impact / dora.deployments.count`. A big time rollup of at least 1 week is recommended to account for the time difference between deployments and when the impact starts.
| `dora.time_to_restore` | distribution | The time in `seconds` between an incident's `started_at` and `finished_at` timestamps.

### Default tags

All default metrics contain the following tags if any are available:

- `service`
- `team`
- `env`
- `repository_id`

**Note**: The `severity` tag is available for `dora.incidents_impact` and `dora.time_to_restore` metrics when it is provided by the failure's data source.

For more information about using `env`, `service`, and `version` tags, see [Getting Started with Tags][6].

## Change lead time metrics

Datadog breaks down change lead time into the following metrics, which represent the different stages from commit creation to deployment.

| Metric | Type | Description |
|---|---|---|
| `dora.time_to_pr_ready` | duration | Time from when the commit is created until the PR is ready for review. This metric is only available for commits that were made before the PR was marked as ready for review. |
| `dora.review_time` | duration | Time from when the PR is marked ready for review until it receives the last approval. This metric is only available for commits that were made before the PR is approved. |
| `dora.merge_time` | duration | Time from the last approval until the PR is merged. |
| `dora.time_to_deploy` | duration | Time from PR merge to start of deployment. If a commit does not have an associated PR, this metric is calculated as the time from commit creation to start of deployment. |
| `dora.deploy_time` | duration | Time from start of deployment to end of deployment. This metric is not available if there is no deployment duration information. |

These metrics are only computed when the source of the repository metadata is GitHub, and there must be a pull request (PR) associated with a commit, if any. A commit is associated with a PR if the commit is first introduced to the target branch when merging that PR. If a commit does not have an associated PR, only `dora.time_to_deploy` and `dora.deploy_time` metrics are available.

**Note:** These metrics are emitted for every commit and not per deployment.

## Examine metrics in Event Management

Default DORA Metrics are available in the [Events Explorer][1]. To search and filter on your DORA Metrics events, navigate to [**Service Management** > **Event Management** > **Explorer**][11] and enter `source:software_delivery_insights` in the search query.

{{< img src="dora_metrics/events.png" alt="Events collected from DORA Metrics in the Events Explorer" style="width:100%;" >}}

These metrics can be queried programmatically by using the [Query timeseries points][5] and [Query timeseries data across multiple products][6] API endpoints with the source `software_delivery_insights`.

## Further Reading

{{< partial name="whats-next/whats-next.html" >}}

[1]: /service_management/events/explorer/
[2]: /api/latest/metrics/#query-timeseries-points
[3]: /api/latest/metrics/#query-timeseries-data-across-multiple-products
[5]: https://app.datadoghq.com/event/explorer?query=source%3Asoftware_delivery_insights
[6]: /getting_started/tagging/
[7]: /api/latest/dora-metrics/
[8]: https://app.datadoghq.com/ci/dora
[9]: https://docs.datadoghq.com/metrics/
[10]: /dora_metrics/deployments/
[11]: https://app.datadoghq.com/event/explorer?query=source%3Asoftware_delivery_insights%20&cols=&messageDisplay=expanded-lg&options=&refresh_mode=sliding&sort=DESC&from_ts=1714391730343&to_ts=1714392630343&live=true
Loading
Loading