[DOCS-8428] Rename setup doc and add referencing files in k8s (#24131)

* rename setup and add referencing files in k8s * add links to sections * Update content/en/observability_pipelines/advanced_configurations.md Co-authored-by: Bryce Eadie <[email protected]> --------- Co-authored-by: Bryce Eadie <[email protected]>
DataDog · Jul 16, 2024 · e24c739 · e24c739
1 parent c3163da
commit e24c739
Show file tree

Hide file tree

Showing 6 changed files with 116 additions and 64 deletions.
diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml
@@ -4194,215 +4194,215 @@ menu:
       identifier: observability_pipelines
       parent: log_management_heading
       weight: 10000
-    - name: Setup
-      url: observability_pipelines/setup_opw/
-      parent: observability_pipelines
-      identifier: observability_pipelines_setup_opw
-      weight: 1
     - name: Log Volume Control
       url: observability_pipelines/log_volume_control/
       parent: observability_pipelines
       identifier: observability_pipelines_log_volume_control
-      weight: 2
+      weight: 1
     - name: Datadog Agent
       url: observability_pipelines/log_volume_control/datadog_agent/
       parent: observability_pipelines_log_volume_control
       identifier: observability_pipelines_log_volume_control_datadog_agent
-      weight: 2001
+      weight: 101
     - name: Fluent
       url: observability_pipelines/log_volume_control/fluent/
       parent: observability_pipelines_log_volume_control
       identifier: observability_pipelines_log_volume_control_fluent
-      weight: 2002
+      weight: 102
     - name: HTTP Client
       url: observability_pipelines/log_volume_control/http_client/
       parent: observability_pipelines_log_volume_control
       identifier: observability_pipelines_log_volume_control_http_client
-      weight: 2003
+      weight: 103
     - name: Splunk HTTP Event Collector
       url: observability_pipelines/log_volume_control/splunk_hec/
       parent: observability_pipelines_log_volume_control
       identifier: observability_pipelines_log_volume_control_splunk_hec
-      weight: 2004
+      weight: 104
     - name: Splunk Forwarders (TCP)
       url: observability_pipelines/log_volume_control/splunk_tcp/
       parent: observability_pipelines_log_volume_control
       identifier: observability_pipelines_log_volume_control_splunk_tcp
-      weight: 2005
+      weight: 105
     - name: Sumo Logic Hosted Collector
       url: observability_pipelines/log_volume_control/sumo_logic_hosted_collector/
       parent: observability_pipelines_log_volume_control
       identifier: observability_pipelines_log_volume_control_sumo_logic_hosted_collector
-      weight: 2006
+      weight: 106
     - name: Syslog
       url: observability_pipelines/log_volume_control/syslog/
       parent: observability_pipelines_log_volume_control
       identifier: observability_pipelines_log_volume_control_syslog
-      weight: 2007
+      weight: 107
     - name: Dual Ship Logs
       url: observability_pipelines/dual_ship_logs/
       parent: observability_pipelines
       identifier: observability_pipelines_dual_ship_logs
-      weight: 3
+      weight: 2
     - name: Datadog Agent
       url: observability_pipelines/dual_ship_logs/datadog_agent/
       parent: observability_pipelines_dual_ship_logs
       identifier: observability_pipelines_dual_ship_logs_datadog_agent
-      weight: 3001
+      weight: 201
     - name: Fluent
       url: observability_pipelines/dual_ship_logs/fluent/
       parent: observability_pipelines_dual_ship_logs
       identifier: observability_pipelines_dual_ship_logs_fluent
-      weight: 3002
+      weight: 202
     - name: HTTP Client
       url: observability_pipelines/dual_ship_logs/http_client/
       parent: observability_pipelines_dual_ship_logs
       identifier: observability_pipelines_dual_ship_logs_http_client
-      weight: 3003
+      weight: 203
     - name: Splunk HTTP Event Collector
       url: observability_pipelines/dual_ship_logs/splunk_hec/
       parent: observability_pipelines_dual_ship_logs
       identifier: observability_pipelines_dual_ship_logs_splunk_hec
-      weight: 3004
+      weight: 204
     - name: Splunk Forwarders (TCP)
       url: observability_pipelines/dual_ship_logs/splunk_tcp/
       parent: observability_pipelines_dual_ship_logs
       identifier: observability_pipelines_dual_ship_logs_splunk_tcp
-      weight: 3005
+      weight: 205
     - name: Sumo Logic Hosted Collector
       url: observability_pipelines/dual_ship_logs/sumo_logic_hosted_collector/
       parent: observability_pipelines_dual_ship_logs
       identifier: observability_pipelines_dual_ship_logs_sumo_logic_hosted_collector
-      weight: 3006
+      weight: 206
     - name: Syslog
       url: observability_pipelines/dual_ship_logs/syslog/
       parent: observability_pipelines_dual_ship_logs
       identifier: observability_pipelines_dual_ship_logs_syslog
-      weight: 3007
+      weight: 207
     - name: Archive Logs
       url: observability_pipelines/archive_logs/
       parent: observability_pipelines
       identifier: observability_pipelines_archive_logs
-      weight: 4
+      weight: 3
     - name: Datadog Agent
       url: observability_pipelines/archive_logs/datadog_agent/
       parent: observability_pipelines_archive_logs
       identifier: observability_pipelines_archive_logs_datadog_agent
-      weight: 4001
+      weight: 301
     - name: Fluent
       url: observability_pipelines/archive_logs/fluent/
       parent: observability_pipelines_archive_logs
       identifier: observability_pipelines_archive_logs_fluent
-      weight: 4002
+      weight: 302
     - name: HTTP Client
       url: observability_pipelines/archive_logs/http_client/
       parent: observability_pipelines_archive_logs
       identifier: observability_pipelines_archive_logs_http_client
-      weight: 4003
+      weight: 303
     - name: Splunk HTTP Event Collector
       url: observability_pipelines/archive_logs/splunk_hec/
       parent: observability_pipelines_archive_logs
       identifier: observability_pipelines_archive_logs_splunk_hec
-      weight: 4004
+      weight: 304
     - name: Splunk Forwarders (TCP)
       url: observability_pipelines/archive_logs/splunk_tcp/
       parent: observability_pipelines_archive_logs
       identifier: observability_pipelines_archive_logs_splunk_tcp
-      weight: 4005
+      weight: 305
     - name: Sumo Logic Hosted Collector
       url: observability_pipelines/archive_logs/sumo_logic_hosted_collector/
       parent: observability_pipelines_archive_logs
       identifier: observability_pipelines_archive_logs_sumo_logic_hosted_collector
-      weight: 4006
+      weight: 306
     - name: Syslog
       url: observability_pipelines/archive_logs/syslog/
       parent: observability_pipelines_archive_logs
       identifier: observability_pipelines_archive_logs_syslog
-      weight: 4007
+      weight: 307
     - name: Split Logs
       url: observability_pipelines/split_logs/
       parent: observability_pipelines
       identifier: observability_pipelines_split_logs
-      weight: 5
+      weight: 4
     - name: Datadog Agent
       url: observability_pipelines/split_logs/datadog_agent/
       parent: observability_pipelines_split_logs
       identifier: observability_pipelines_split_logs_datadog_agent
-      weight: 5001
+      weight: 401
     - name: Fluent
       url: observability_pipelines/split_logs/fluent/
       parent: observability_pipelines_split_logs
       identifier: observability_pipelines_split_logs_fluent
-      weight: 5002
+      weight: 402
     - name: HTTP Client
       url: observability_pipelines/split_logs/http_client/
       parent: observability_pipelines_split_logs
       identifier: observability_pipelines_split_logs_http_client
-      weight: 5003
+      weight: 403
     - name: Splunk HTTP Event Collector
       url: observability_pipelines/split_logs/splunk_hec/
       parent: observability_pipelines_split_logs
       identifier: observability_pipelines_split_logs_splunk_hec
-      weight: 5004
+      weight: 404
     - name: Splunk Forwarders (TCP)
       url: observability_pipelines/split_logs/splunk_tcp/
       parent: observability_pipelines_split_logs
       identifier: observability_pipelines_split_logs_splunk_tcp
-      weight: 5005
+      weight: 405
     - name: Sumo Logic Hosted Collector
       url: observability_pipelines/split_logs/sumo_logic_hosted_collector/
       parent: observability_pipelines_split_logs
       identifier: observability_pipelines_split_logs_sumo_logic_hosted_collector
-      weight: 5006
+      weight: 406
     - name: Syslog
       url: observability_pipelines/split_logs/syslog/
       parent: observability_pipelines_split_logs
       identifier: observability_pipelines_split_logs_syslog
-      weight: 5007
+      weight: 407
     - name: Sensitive Data Redaction
       url: observability_pipelines/sensitive_data_redaction/
       parent: observability_pipelines
       identifier: observability_pipelines_sensitive_data_redaction
-      weight: 6
+      weight: 5
     - name: Datadog Agent
       url: observability_pipelines/sensitive_data_redaction/datadog_agent/
       parent: observability_pipelines_sensitive_data_redaction
       identifier: observability_pipelines_sensitive_data_redaction_datadog_agent
-      weight: 6001
+      weight: 501
     - name: Fluent
       url: observability_pipelines/sensitive_data_redaction/fluent/
       parent: observability_pipelines_sensitive_data_redaction
       identifier: observability_pipelines_sensitive_data_redaction_fluent
-      weight: 6002
+      weight: 502
     - name: HTTP Client
       url: observability_pipelines/sensitive_data_redaction/http_client/
       parent: observability_pipelines_sensitive_data_redaction
       identifier: observability_pipelines_sensitive_data_redaction_http_client
-      weight: 6003
+      weight: 503
     - name: Splunk HTTP Event Collector
       url: observability_pipelines/sensitive_data_redaction/splunk_hec/
       parent: observability_pipelines_sensitive_data_redaction
       identifier: observability_pipelines_sensitive_data_redaction_splunk_hec
-      weight: 6004
+      weight: 504
     - name: Splunk Forwarders (TCP)
       url: observability_pipelines/sensitive_data_redaction/splunk_tcp/
       parent: observability_pipelines_sensitive_data_redaction
       identifier: observability_pipelines_sensitive_data_redaction_splunk_tcp
-      weight: 6005
+      weight: 505
     - name: Sumo Logic Hosted Collector
       url: observability_pipelines/sensitive_data_redaction/sumo_logic_hosted_collector/
       parent: observability_pipelines_sensitive_data_redaction
       identifier: observability_pipelines_sensitive_data_redaction_sumo_logic_hosted_collector
-      weight: 6006
+      weight: 506
     - name: Syslog
       url: observability_pipelines/sensitive_data_redaction/syslog/
       parent: observability_pipelines_sensitive_data_redaction
       identifier: observability_pipelines_sensitive_data_redaction_syslog
-      weight: 6007
+      weight: 507
     - name: Update Existing Pipelines
       url: observability_pipelines/update_existing_pipelines/
       parent: observability_pipelines
       identifier: observability_pipelines_update_existing_pipelines
+      weight: 6
+    - name: Advanced Configurations
+      url: observability_pipelines/advanced_configurations/
+      parent: observability_pipelines
+      identifier: observability_pipelines_advanced_configurations
       weight: 7
     - name: Best Practices for Scaling Observability Pipelines
       url: observability_pipelines/best_practices_for_scaling_observability_pipelines/

diff --git a/content/en/observability_pipelines/_index.md b/content/en/observability_pipelines/_index.md
@@ -57,24 +57,25 @@ The Datadog UI provides a control plane to manage your Observability Pipelines W
 
 ## Get started
 
-1. [Set up bootstrap options for the Observability Pipelines Worker][1].
-1. Navigate to [Observability Pipelines][2].
+1. Navigate to [Observability Pipelines][1].
 1. Select a use case:
-    - [Log volume control][3]
-    - [Dual ship logs][4]
-    - [Split logs][5]
-    - [Archive logs to Datadog Archives][6]
-    - [Sensitive data redaction][7]
+    - [Log volume control][2]
+    - [Dual ship logs][3]
+    - [Split logs][4]
+    - [Archive logs to Datadog Archives][5]
+    - [Sensitive data redaction][6]
 1. Enable monitors.
 
+See [Advanced Configurations][7] for bootstrapping options and for details on setting up the Worker with Kubernetes.
+
 ## Further Reading
 
 {{< partial name="whats-next/whats-next.html" >}}
 
-[1]: /observability_pipelines/setup_opw/
-[2]: https://app.datadoghq.com/observability-pipelines
-[3]: /observability_pipelines/log_volume_control/
-[4]: /observability_pipelines/dual_ship_logs/
-[5]: /observability_pipelines/split_logs/
-[6]: /observability_pipelines/archive_logs/
-[7]: /observability_pipelines/sensitive_data_redaction/
+[1]: https://app.datadoghq.com/observability-pipelines
+[2]: /observability_pipelines/log_volume_control/
+[3]: /observability_pipelines/dual_ship_logs/
+[4]: /observability_pipelines/split_logs/
+[5]: /observability_pipelines/archive_logs/
+[6]: /observability_pipelines/sensitive_data_redaction/
+[7]: /observability_pipelines/setup_opw/
diff --git a/...t/en/observability_pipelines/setup_opw.md → ...lity_pipelines/advanced_configurations.md b/...t/en/observability_pipelines/setup_opw.md → ...lity_pipelines/advanced_configurations.md
@@ -1,6 +1,8 @@
 ---
-title: Setup
+title: Advanced Configurations
 disable_toc: false
+aliases:
+  - /observability_pipelines/setup_opw/
 further_reading:
 - link: "/observability_pipelines/log_volume_control/"
   tag: "Documentation"
@@ -24,6 +26,10 @@ further_reading:
 
 ## Overview
 
+This document goes over [bootstrapping the Observability Pipelines Worker](#bootstrap-options) and [referencing files in Kubernetes](#referencing-files-in-kubernetes).
+
+## Bootstrap Options
+
 <div class="alert alert-warning">All configuration file paths specified in the pipeline need to be under <code>DD_OP_DATA_DIR/config</code>.
 Modifying files under that location while OPW is running might have adverse effects.
 </div>
@@ -34,8 +40,6 @@ Bootstrap the Observability Pipelines Worker within your infrastructure before y
 - Bootstrap file: `/etc/observability-pipelines-worker/bootstrap.yaml`
 - Environment variables file: `/etc/default/observability-pipelines-worker`
 
-## Bootstrap Options
-
 To set bootstrap options, do one of the following:
 - Use environmental variables.
 - Create a `bootstrap.yaml` and start the Worker instance with `--bootstrap-config /path/to/bootstrap.yaml`.
@@ -73,10 +77,49 @@ To set bootstrap options, do one of the following:
 <br>&nbsp;&nbsp;&nbsp;1. `DD_PROXY_HTTP(S)`
 <br>&nbsp;&nbsp;&nbsp;2. `HTTP(S)_PROXY`
 <br>&nbsp;&nbsp;&nbsp;3. `proxy`
-: 
+:
 : An example proxy configuration:
 : &nbsp;&nbsp;&nbsp;&nbsp;proxy:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;enabled: true<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;https: https://foo.bar:3128
 : <b>Note</b>: The `DD_PROXY_HTTP(S)` and `HTTP(S)_PROXY` environment variables need to be already exported in your environment for the Worker to resolve them. They cannot be prepended to the Worker installation script.
+
+## Referencing files in Kubernetes
+
+If you are referencing files in Kubernetes for Google Cloud Storage authentication, TLS certificates for certain sources, or an enrichment table processor, you need to use `volumeMounts[*].subPath` to mount files from a `configMap` or `secret`.
+
+For example, if you have a `secret` defined as:
+
+```
+apiVersion: v1
+kind: Secret
+metadata:
+  name: my-secret
+type: Opaque
+data:
+  credentials1.json: bXktc2VjcmV0LTE=
+  credentials2.json: bXktc2VjcmV0LTI=
+```
+
+Then you need to override `extraVolumes` and `extraVolumeMounts` in the `values.yaml` file to mount the secret files to Observability Pipelines Worker pods using `subPath`:
+
+```
+# extraVolumes -- Specify additional Volumes to use.
+extraVolumes:
+  - name: my-secret-volume
+    secret:
+      secretName: my-secret
+
+# extraVolumeMounts -- Specify Additional VolumeMounts to use.
+extraVolumeMounts:
+  - name: my-secret-volume
+    mountPath: /var/lib/observability-pipelines-worker/config/credentials1.json
+    subPath: credentials1.json
+  - name: my-secret-volume
+    mountPath: /var/lib/observability-pipelines-worker/config/credentials2.json
+    subPath: credentials2.json
+```
+
+**Note**: If you override the`datadog.dataDir` parameter, you need to override the `mountPath` as well.
+
 ## Further reading
 
 {{< partial name="whats-next/whats-next.html" >}}

diff --git a/...ervability_pipelines/configure_log_archive/google_cloud_storage/instructions.md b/...ervability_pipelines/configure_log_archive/google_cloud_storage/instructions.md
@@ -12,6 +12,8 @@
 1. Create a Google Cloud Storage [service account][9092].
 1. Follow these [instructions][9093] to create a service account key and download the JSON service account key file. This is the credentials JSON file and must be placed under `DD_OP_DATA_DIR/config`.
 
+**Note**: If you are installing the Worker in Kubernetes, see [Referencing files in Kubernetes][9097] for information on how to reference the credentials file.
+
 ### Connect the storage bucket to Datadog Log Archives
 
 1. Navigate to Datadog [Log Forwarding][9094].
@@ -34,3 +36,4 @@ See the [Log Archives documentation][9096] for additional information.
 [9094]: https://app.datadoghq.com/logs/pipelines/log-forwarding
 [9095]: /logs/log_configuration/archives/?tab=awss3#advanced-settings
 [9096]: /logs/log_configuration/archives
+[9097]: /observability_pipelines/advanced_configurations/#referencing-files-in-kubernetes