Skip to content

Commit

Permalink
[DOCS-8587] Update Obs Pipelines landing doc (#24424)
Browse files Browse the repository at this point in the history
* update landing doc

* edits

* update archive logs

* small edits

* update archive logs

* small edits

* add alt text

* Apply suggestions from code review

Co-authored-by: Jen Gilbert <[email protected]>

* add log enrichment

---------

Co-authored-by: Jen Gilbert <[email protected]>
  • Loading branch information
maycmlee and jhgilbert authored Aug 5, 2024
1 parent d19f067 commit b9c772e
Show file tree
Hide file tree
Showing 2 changed files with 62 additions and 12 deletions.
74 changes: 62 additions & 12 deletions content/en/observability_pipelines/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ further_reading:
text: "Sensitive Data Scanner"
- link: "/agent/configuration/dual-shipping/#yaml-configuration"
tag: "documentation"
text: "Dual ship to Observability Pipelines with the Datadog Agent"
text: "Dual shipping with Observability Pipelines"
- link: "https://www.datadoghq.com/blog/observability-pipelines-sensitive-data-redaction/"
tag: "blog"
text: "Redact sensitive data from your logs on-prem by using Observability Pipelines"
Expand Down Expand Up @@ -43,32 +43,82 @@ Datadog recommends you update Observability Pipelines Worker (OPW) with every mi

{{< img src="observability_pipelines/op_marketecture_04182024.png" alt="A graphic showing different data sources on the left that flows into three hexagons named transform, reduce, and route, with arrows pointing to different destinations for the modified data" style="width:100%;" >}}

Observability Pipelines allows you to collect, process, and route logs in your own infrastructure. It comes with out-of-the-box templates so that you can easily build and deploy pipelines. The templates are purpose-built for the following use cases:
Observability Pipelines allows you to collect, process, and route logs in your own infrastructure. It comes with out-of-the-box [templates](#start-building-pipelines-with-out-of-the-box-templates) so that you can easily build and deploy pipelines.

- Log volume control: Cut down on your log volume before it leaves your infrastructure or network.
- Dual ship logs: Send copies of your logs to multiple destinations.
- Split logs: Send your logs to different destinations based on your use case. For example, you can send DevOps logs to Datadog and security logs to a security vendor.
- Archive logs: Send logs to a log vendor and to an archive in Datadog rehydratable format.
- Sensitive data redaction: Remove sensitive data from your logs before they are routed outside of your infrastructure.
The Observability Pipelines Worker is the software that runs in your infrastructure. It centrally aggregates, processes, and routes your logs based on your use case. This means you can redact sensitive data, pre-process logs, and determine which destinations they should go to, all before the logs leave your environment.

The Observability Pipelines Worker is the software that runs in your infrastructure. It aggregates and centrally processes and routes your logs based on the selected use case.

The Datadog UI provides a control plane to manage your Observability Pipelines Workers. You can build and edit pipelines, deploy pipeline changes to your Workers, and monitor your pipelines to evaluate the health of your pipelines.
The Observability Pipelines UI provides a control plane to manage your Observability Pipelines Workers. You build and edit pipelines and deploy pipeline changes to your Workers from there. You can also enable out-of-the-box monitors for your pipelines so that you can evaluate their health.

## Get started

To set up a pipeline:

1. Navigate to [Observability Pipelines][1].
1. Select a use case:
1. Select a template:
- [Log volume control][2]
- [Dual ship logs][3]
- [Split logs][4]
- [Archive logs to Datadog Archives][5]
- [Sensitive data redaction][6]
- [Log Enrichment][7]
1. Enable monitors.
1. Select and set up your source.
1. Select and set up your destinations.
1. Set up you processors.
1. Install the Observability Pipelines Worker.
1. Enable monitors for your pipeline.

See [Advanced Configurations][8] for bootstrapping options and for details on setting up the Worker with Kubernetes.

## Explore Observability Pipelines

### Build pipelines with out-of-the-box templates

{{< img src="observability_pipelines/templates.png" alt="The Observability Pipelines UI showing the six templates" style="width:100%;" >}}

The [templates](#out-of-the-box-templates) are built for the following use cases:

#### Log Volume Control

Raw logs are noisy, and only some logs are useful for further search and analysis during investigations. Use the Log Volume Control template to determine which logs to send to your indexed solution, such as a SIEM or log management solution. This helps you to increase the value of your indexed logs and also remain within your planned budget.

#### Dual Ship Logs

As your organization grows, your observability needs for different use cases, such as security, archiving, and log management, also change. This could mean having to trial different archiving, SIEM, and log management solutions. However, managing log pipelines to different solutions can be complicated. Use the Dual Ship Logs template to centrally aggregate, process, and send copies of your logs to different destinations.

#### Archive Logs

Use the Archive Logs template to store logs in a cloud storage solution (Amazon S3, Google Cloud Storage, or Azure Storage). The archived logs are stored in a Datadog-rehydratable format, so that they can be rehydrated in Datadog as needed. This is useful when:

- You have a high volume of noisy logs, but might need to index them in Datadog Log Management ad hoc for an investigation.
- You are migrating to Datadog Log Management and want to have historical logs after completing the migration.
- You have a retention policy to fulfill compliance requirements but don't necessarily need to index those logs.

#### Split Logs

When you have logs from different services and applications, you might need to send them to different downstream services for querying, analysis, and alerting. For example, you might want to send security logs to a SIEM solution and DevOps logs to Datadog. Use the Split Logs template to preprocess your logs separately for each destination before sending them downstream.

#### Sensitive Data Redaction

Use the Sensitive Data Redaction template to detect and redact sensitive information on premises. The Observability Pipelines sensitive data scanner processor provides 70 out-of-the-box scanning rules, but you can also create your own custom scanning rules using regular expressions. The OOTB rules recognize standard patterns such as credit card numbers, email addresses, IP addresses, API and SSH keys, and access tokens.

#### Log Enrichment

Your organization's different services, systems, and applications all generate logs containing layers of information and in different formats. This can make it difficult to extract the data you need when searching and analyzing the data for an investigation. Use the Log Enrichment template to standardize your logs and enrich them with information, such as data from a reference table.

### Build pipelines in the Observability Pipelines UI

{{% observability_pipelines/use_case_images/log_volume_control %}}

Build your pipelines in the Observability Pipelines UI. After you select one of the out-the-box templates, the onboarding workflow walks you through setting up your source, processors, and destinations. The installation page provides instructions on how to install the Worker in your environment (Docker, Kubernetes, Linux, or CloudFormation).

### Enable out-of-the-box monitors for your pipeline components

After you create your pipeline, enable out-of-the box monitors to get alerted when:

- There are increasing error rates for a component. This could happen because the component is processing data in unexpected formats.
- The Observability Pipelines Worker has high CPU usage or memory usage.
- There are spikes in data dropped by a component.

## Further Reading

{{< partial name="whats-next/whats-next.html" >}}
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit b9c772e

Please sign in to comment.