Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS-7217/7054] Add OP landing and setup docs #22171

Merged
merged 91 commits into from
Apr 11, 2024
Merged
Show file tree
Hide file tree
Changes from 73 commits
Commits
Show all changes
91 commits
Select commit Hold shift + click to select a range
1228c26
add dual ship splunk
maycmlee Feb 28, 2024
0355a6e
add destination variables and other edits
maycmlee Mar 4, 2024
af4219c
updates
maycmlee Mar 5, 2024
3b3ebf2
Merge branch 'may/obs-pipelines-2.0' into may/op-2.0-use-cases
maycmlee Mar 11, 2024
bdaca93
add content
maycmlee Mar 11, 2024
c393df4
add more installation content
maycmlee Mar 11, 2024
1ef3cc8
fix typos
maycmlee Mar 11, 2024
83a22fb
another typo
maycmlee Mar 11, 2024
a735a67
add landing page
maycmlee Mar 11, 2024
7e49424
add update existing pipelines
maycmlee Mar 12, 2024
d4a950c
add existing doc as separate doc
maycmlee Mar 12, 2024
be1e77b
small edits
maycmlee Mar 12, 2024
b6f2d0c
update to configure existing pipelines
maycmlee Mar 13, 2024
435c1c2
Merge branch 'may/obs-pipelines-2.0' into may/op-2.0-use-cases
maycmlee Mar 13, 2024
39be809
add migration note
maycmlee Mar 13, 2024
218171b
edit processor
maycmlee Mar 13, 2024
6f9b809
add docker example
maycmlee Mar 13, 2024
d121a23
small edit
maycmlee Mar 13, 2024
0c4eb1b
update existing pipelines
maycmlee Mar 13, 2024
2fe5c5d
move configure existing to separate pr
maycmlee Mar 14, 2024
44f7c20
Merge branch 'may/obs-pipelines-2.0' into may/op-2.0-use-cases
maycmlee Mar 25, 2024
efffa7b
updates
maycmlee Mar 25, 2024
ef8e2b6
reorder nav and small edits
maycmlee Mar 25, 2024
97d5089
update redaction
maycmlee Mar 25, 2024
c8a2786
update landing page
maycmlee Mar 27, 2024
deb10bd
small edits include rc update
maycmlee Mar 27, 2024
b096b41
update filenames
maycmlee Mar 28, 2024
e52cad5
add datadog
maycmlee Mar 28, 2024
6c672e4
add use case indexes
maycmlee Mar 28, 2024
a9cbcd7
small edits
maycmlee Mar 28, 2024
7a05af9
update landing pages
maycmlee Mar 29, 2024
a8fb2e3
add architecture
maycmlee Mar 29, 2024
ca2eb13
edits to landing pages
maycmlee Apr 1, 2024
1a92c1c
move datadog to top
maycmlee Apr 2, 2024
77efc35
add reuse shortcode files
maycmlee Apr 2, 2024
644f74f
add all nav items
maycmlee Apr 3, 2024
0ac9e9d
add splunk hec pages
maycmlee Apr 3, 2024
44e868d
update nav and titles and more
maycmlee Apr 3, 2024
b97c4eb
more updates to names
maycmlee Apr 4, 2024
9915581
add prereqs, source settings, desintation settings shortcodes
maycmlee Apr 4, 2024
9920ec5
add processors
maycmlee Apr 4, 2024
98391f2
add worker installation shortcodes
maycmlee Apr 4, 2024
2490766
add log source config shortcodes
maycmlee Apr 4, 2024
56019d9
remove old splunck hec doc
maycmlee Apr 4, 2024
ca4ff5a
add archive log shortcodes
maycmlee Apr 4, 2024
2593f14
add sds and datadog archives
maycmlee Apr 4, 2024
990351c
update prereqs
maycmlee Apr 4, 2024
fbe7292
update shortcodes
maycmlee Apr 4, 2024
29758d8
add upgrade div and legacy warning shortcodes
maycmlee Apr 4, 2024
d4e8283
add processors shortcode
maycmlee Apr 4, 2024
17f63fb
splunk tcp docs
maycmlee Apr 4, 2024
f64c7f9
fix links
maycmlee Apr 5, 2024
d2e6cd1
really fix all shortcode links
maycmlee Apr 5, 2024
bccf5ac
update all splunk tcp docs
maycmlee Apr 5, 2024
bdd1db9
add archives dest tcp
maycmlee Apr 5, 2024
9eb770a
add sumo logic docs
maycmlee Apr 5, 2024
1baa773
fix sumo logic docs
maycmlee Apr 5, 2024
d8ba4e7
add datadog agent docs
maycmlee Apr 5, 2024
b15c46a
make titles consistent
maycmlee Apr 5, 2024
701774b
consistency edits
maycmlee Apr 5, 2024
2aa99f3
add dd overviews
maycmlee Apr 5, 2024
a2de920
add cloudformation table
maycmlee Apr 5, 2024
8a8a8a5
fix landing page links
maycmlee Apr 5, 2024
b9d588e
add path notation example
maycmlee Apr 5, 2024
c6f848a
small fix
maycmlee Apr 5, 2024
7185ff4
add splt logs prereqs tabs
maycmlee Apr 5, 2024
f755a83
add gov note
maycmlee Apr 5, 2024
511b6c6
small edits
maycmlee Apr 5, 2024
9b9c44d
small edits
maycmlee Apr 5, 2024
ede7e70
small edits
maycmlee Apr 5, 2024
bd912a3
Apply suggestions from code review
maycmlee Apr 5, 2024
6d3f526
Apply suggestions from code review
maycmlee Apr 8, 2024
ad26c00
capitalize placeholders and small edits
maycmlee Apr 9, 2024
036b304
Update content/en/observability_pipelines/_index.md
maycmlee Apr 9, 2024
f7defe5
[DOCS-7432] Configure existing pipelines for OP (#22205)
maycmlee Apr 9, 2024
dc41b06
remove sources list and add further reading to landing
maycmlee Apr 10, 2024
f188eae
add split logs images
maycmlee Apr 10, 2024
9a44d26
add alt text and general proc image
maycmlee Apr 10, 2024
7c3c426
try to fix image
maycmlee Apr 10, 2024
f3a0d08
try again to fix
maycmlee Apr 10, 2024
7441a61
desperately try to fix images
maycmlee Apr 10, 2024
2c26ff8
add images to md
maycmlee Apr 10, 2024
592cb3b
add images
maycmlee Apr 10, 2024
00d4f52
update images
maycmlee Apr 10, 2024
f29018c
remap to edit fields
maycmlee Apr 10, 2024
777d42b
remove datadog tab from destination env var
maycmlee Apr 10, 2024
58ea604
make us1-fed note red
maycmlee Apr 10, 2024
f631461
cloudformation contact support
maycmlee Apr 10, 2024
55edc0b
update config existing pipelines title and add links to that doc
maycmlee Apr 10, 2024
10792c1
update nav and architecture nav, title, and filename
maycmlee Apr 10, 2024
7def59f
Update content/en/observability_pipelines/log_volume_control/_index.md
maycmlee Apr 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
130 changes: 130 additions & 0 deletions config/_default/menus/menus.en.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3352,6 +3352,136 @@ main:
identifier: observability_pipelines
parent: log_management_heading
weight: 10000
- name: Log Volume Control
url: observability_pipelines/log_volume_control/
parent: observability_pipelines
identifier: observability_pipelines_log_volume_control
weight: 1
- name: Splunk HTTP Event Collector
url: observability_pipelines/log_volume_control/splunk_hec/
parent: observability_pipelines_log_volume_control
identifier: observability_pipelines_log_volume_control_splunk_hec
weight: 1001
- name: Splunk Forwarders (TCP)
url: observability_pipelines/log_volume_control/splunk_tcp/
parent: observability_pipelines_log_volume_control
identifier: observability_pipelines_log_volume_control_splunk_tcp
weight: 1002
- name: Sumo Logic Hosted Collector
url: observability_pipelines/log_volume_control/sumo_logic_hosted_collector/
parent: observability_pipelines_log_volume_control
identifier: observability_pipelines_log_volume_control_sumo_logic_hosted_collector
weight: 1003
- name: Dual Ship Logs
url: observability_pipelines/dual_ship_logs/
parent: observability_pipelines
identifier: observability_pipelines_dual_ship_logs
weight: 2
- name: Datadog Agent
url: observability_pipelines/dual_ship_logs/datadog_agent/
parent: observability_pipelines_dual_ship_logs
identifier: observability_pipelines_dual_ship_logs_datadog_agent
weight: 2001
- name: Splunk HTTP Event Collector
url: observability_pipelines/dual_ship_logs/splunk_hec/
parent: observability_pipelines_dual_ship_logs
identifier: observability_pipelines_dual_ship_logs_splunk_hec
weight: 2002
- name: Splunk Forwarders (TCP)
url: observability_pipelines/dual_ship_logs/splunk_tcp/
parent: observability_pipelines_dual_ship_logs
identifier: observability_pipelines_dual_ship_logs_splunk_tcp
weight: 2003
- name: Sumo Logic Hosted Collector
url: observability_pipelines/dual_ship_logs/sumo_logic_hosted_collector/
parent: observability_pipelines_dual_ship_logs
identifier: observability_pipelines_dual_ship_logs_sumo_logic_hosted_collector
weight: 2004
- name: Archive Logs
url: observability_pipelines/archive_logs/
parent: observability_pipelines
identifier: observability_pipelines_archive_logs
weight: 3
- name: Datadog Agent
url: observability_pipelines/archive_logs/datadog_agent/
parent: observability_pipelines_archive_logs
identifier: observability_pipelines_archive_logs_datadog_agent
weight: 3001
- name: Splunk HTTP Event Collector
url: observability_pipelines/archive_logs/splunk_hec/
parent: observability_pipelines_archive_logs
identifier: observability_pipelines_archive_logs_splunk_hec
weight: 3002
- name: Splunk Forwarders (TCP)
url: observability_pipelines/archive_logs/splunk_tcp/
parent: observability_pipelines_archive_logs
identifier: observability_pipelines_archive_logs_splunk_tcp
weight: 3003
- name: Sumo Logic Hosted Collector
url: observability_pipelines/archive_logs/sumo_logic_hosted_collector/
parent: observability_pipelines_archive_logs
identifier: observability_pipelines_archive_logs_sumo_logic_hosted_collector
weight: 3004
- name: Split Logs
url: observability_pipelines/split_logs/
parent: observability_pipelines
identifier: observability_pipelines_split_logs
weight: 4
- name: Datadog Agent
url: observability_pipelines/split_logs/datadog_agent/
parent: observability_pipelines_split_logs
identifier: observability_pipelines_split_logs_datadog_agent
weight: 4001
- name: Splunk HTTP Event Collector
url: observability_pipelines/split_logs/splunk_hec/
parent: observability_pipelines_split_logs
identifier: observability_pipelines_split_logs_splunk_hec
weight: 4002
- name: Splunk Forwarders (TCP)
url: observability_pipelines/split_logs/splunk_tcp/
parent: observability_pipelines_split_logs
identifier: observability_pipelines_split_logs_splunk_tcp
weight: 4003
- name: Sumo Logic Hosted Collector
url: observability_pipelines/split_logs/sumo_logic_hosted_collector/
parent: observability_pipelines_split_logs
identifier: observability_pipelines_split_logs_sumo_logic_hosted_collector
weight: 4004
- name: Sensitive Data Redaction
url: observability_pipelines/sensitive_data_redaction/
parent: observability_pipelines
identifier: observability_pipelines_sensitive_data_redaction
weight: 5
- name: Datadog Agent
url: observability_pipelines/sensitive_data_redaction/datadog_agent/
parent: observability_pipelines_sensitive_data_redaction
identifier: observability_pipelines_sensitive_data_redaction_datadog_agent
weight: 5001
- name: Splunk HTTP Event Collector
url: observability_pipelines/sensitive_data_redaction/splunk_hec/
parent: observability_pipelines_sensitive_data_redaction
identifier: observability_pipelines_sensitive_data_redaction_splunk_hec
weight: 5002
- name: Splunk Forwarders (TCP)
url: observability_pipelines/sensitive_data_redaction/splunk_tcp/
parent: observability_pipelines_sensitive_data_redaction
identifier: observability_pipelines_sensitive_data_redaction_splunk_tcp
weight: 5003
- name: Sumo Logic Hosted Collector
url: observability_pipelines/sensitive_data_redaction/sumo_logic_hosted_collector/
parent: observability_pipelines_sensitive_data_redaction
identifier: observability_pipelines_sensitive_data_redaction_sumo_logic_hosted_collector
weight: 5004
- name: Configure Existing Pipelines
url: observability_pipelines/configure_existing_pipelines/
parent: observability_pipelines
identifier: observability_pipelines_configure_existing_pipelines
weight: 6
- name: Architecture
url: observability_pipelines/architecture/
parent: observability_pipelines
identifier: observability_pipelines_architecture
weight: 7
- name: Log Management
url: logs/
pre: log
Expand Down
2 changes: 1 addition & 1 deletion content/en/agent/remote_config/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ The following products and features are supported with Remote Configuration:
- **Automatic deployment of custom Agent rules**: Automatically deploy your custom Agent rules to designated hosts (all hosts or a defined subset of hosts).

### Observability Pipelines
<div class="alert alert-info">This feature is in private beta.</div>
<div class="alert alert-info">This feature is in public beta.</div>

- **Remotely deploy and update [Observability Pipelines Workers][4] (OPW)**: Build and edit pipelines in the Datadog UI, rolling out your configuration changes to OPW instances running in your environment.

Expand Down
52 changes: 52 additions & 0 deletions content/en/observability_pipelines/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
---
title: Observability Pipelines
kind: documentation
disable_toc: false
---

{{< site-region region="gov" >}}
<div class="alert alert-warning">Observability Pipelines is not available on the US1-FED Datadog site.</div>
{{< /site-region >}}

<div class="alert alert-info">
Datadog recommends you update Observability Pipelines Worker (OPW) with every minor and patch release, or, at a minimum, monthly. <br><br> Upgrading to a major OPW version and keeping it updated is the only supported way to get the latest OPW functionality, fixes, and security updates.
</div>

## Overview

Observability Pipelines allows you to collect, process, and route logs in your own infrastructure. It comes with out-of-the-box templates so that you can easily build and deploy pipelines. The templates are purpose-built for the following use cases:

- Log volume control: Cut down on your log volume before it leaves your infrastructure or network.
- Dual ship logs: Send copies of your logs to multiple destinations.
- Split logs: Send your logs to different destinations based on your use case. For example, you can send DevOps logs to Datadog and security logs to a security vendor.
- Archive logs: Send logs to a log vendor and to an archive in Datadog rehydratable format.
- Sensitive data redaction: Redact sensitive data in your logs before they are routed outside of your infrastructure.
maycmlee marked this conversation as resolved.
Show resolved Hide resolved

The Observability Pipelines Worker is the software that runs in your infrastructure. It aggregates and centrally processes and routes your logs based on the selected use case.

The Datadog UI provides a control plane to manage your Observability Pipelines Workers. You can build and edit pipelines, deploy pipeline changes to your Workers, and monitor your pipelines to evaluate the health of your pipelines.

The following sources are available:

- Datadog Agent
- Splunk HTTP Event Collector (HEC)
- Splunk Heavy and Universal Forwarders (TCP)
- Sumo Logic Hosted Collector

## Get started

1. Navigate to [Observability Pipelines][1].
1. Select a use case:
- [Log volume control][2]
- [Dual ship logs][3]
- [Split logs][4]
- [Archive logs to Datadog Archives][5]
- [Sensitive data redaction][6]
1. Enable monitors.

[1]: https://app.datadoghq.com/observability-pipelines
[2]: /observability_pipelines/log_volume_control/
[3]: /observability_pipelines/dual_ship_logs/
[4]: /observability_pipelines/split_logs/
[5]: /observability_pipelines/archive_logs/
[6]: /observability_pipelines/sensitive_data_redaction/
Loading
Loading