[DOCS-7217/7054] Add OP landing and setup docs #22171
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
maycmlee
changed the title
maycmlee
commented
Apr 8, 2024
layouts/shortcodes/observability_pipelines/processors/dedupe.md
Outdated
Show resolved
Hide resolved
Co-authored-by: Jen Gilbert <[email protected]>
jhgilbert
approved these changes
Apr 9, 2024
Co-authored-by: Jen Gilbert <[email protected]>
* add doc * small edits * add source env vars shortcode and update doc * update overview * update shortcode content * update shortcodes * small edit * fix link * Apply suggestions from code review Co-authored-by: Jen Gilbert <[email protected]> * Update layouts/shortcodes/observability_pipelines/configure_existing_pipelines/destination_env_vars/sumo_logic.md Co-authored-by: Jen Gilbert <[email protected]> * fix amazon eks shortcode * uppercase placeholders * small edits and fixes --------- Co-authored-by: Jen Gilbert <[email protected]>
pront
reviewed
Apr 10, 2024
|
|
||
| As your infrastructure and applications grow, so does your log volume and the complexity of the data. A large volume of logs can introduce a lot of noise and make it difficult to analyze and troubleshoot logs. Use Observability Pipelines's processors to decide which logs are valuable and which ones are noisy and uninteresting, before sending your logs to their destinations. You can use the following processors in the Observability Pipeline Worker to manage your logs: | ||
|
|
||
| - Filter: Add a query to send only a subset of logs based on your conditions. |
There was a problem hiding this comment.
Should this be highlighted a little bit more? Maybe a bold font?
pront
reviewed
Apr 10, 2024
|
|
||
| - Filter: Add a query to send only a subset of logs based on your conditions. | ||
| - Sample: Define a sampling rate to send only a subset of your logs. | ||
| - Quota: Enforce soft or hard daily limits on either the volume of log data or the number of log events. |
There was a problem hiding this comment.
What does "soft or hard" mean here? Perhaps it would be simpler to just say "daily limits".
There was a problem hiding this comment.
Removed "soft and hard"
There was a problem hiding this comment.
Hi @pront! I think Barry was referring to soft quotas where you'd set up a monitor to alert you but don't do anything to the data ingested after the limit. See: https://docs.datadoghq.com/observability_pipelines/guide/set_quotas_for_data_sent_to_a_destination/#overview:~:text=You%20can%20also,to%20another%20destination.
You can also use multiple quota transforms to set different warning and alert limits. For example, use the first quota transform to set a limit at a warning level and send a warning notification when the limit is reached. Then, use a second quota transform to set a hard limit. When the hard limit is reached, you can send out an alert notification and start sampling the data sent after the limit or reroute that data to another destination.
But let's keep it simple for now. We can add that information in later. Thanks!
maycmlee
commented
Apr 11, 2024
content/en/observability_pipelines/log_volume_control/_index.md
Outdated
Show resolved
Hide resolved
|
|
||
| As your infrastructure and applications grow, so does your log volume and the complexity of the data. A large volume of logs can introduce a lot of noise and make it difficult to analyze and troubleshoot logs. Use Observability Pipelines's processors to decide which logs are valuable and which ones are noisy and uninteresting, before sending your logs to their destinations. You can use the following processors in the Observability Pipeline Worker to manage your logs: | ||
|
|
||
| - Filter: Add a query to send only a subset of logs based on your conditions. |
|
|
||
| - Filter: Add a query to send only a subset of logs based on your conditions. | ||
| - Sample: Define a sampling rate to send only a subset of your logs. | ||
| - Quota: Enforce soft or hard daily limits on either the volume of log data or the number of log events. |
There was a problem hiding this comment.
Removed "soft and hard"
1 task
jhgilbert
added a commit
that referenced
this pull request
Apr 18, 2024
* hide legacy docs * fix typo * update legacy doc links to legacy docs * add legacy to titles * fix dupe * add legacy to config title * add legacy param (#22130) * [PRODUCTSA-1093] Refine K8S Helm chart values (#21811) * add alert * add aliases * remove alias for monitoring * add warning to pages * Update content/en/observability_pipelines/legacy/_index.md Co-authored-by: Esther Kim <[email protected]> * [DOCS-7217/7054] Add OP landing and setup docs (#22171) * add dual ship splunk * add destination variables and other edits * updates * add content * add more installation content * fix typos * another typo * add landing page * add update existing pipelines * add existing doc as separate doc * small edits * update to configure existing pipelines * add migration note * edit processor * add docker example * small edit * update existing pipelines * move configure existing to separate pr * updates * reorder nav and small edits * update redaction * update landing page * small edits include rc update * update filenames * add datadog * add use case indexes * small edits * update landing pages * add architecture * edits to landing pages * move datadog to top * add reuse shortcode files * add all nav items * add splunk hec pages * update nav and titles and more * more updates to names * add prereqs, source settings, desintation settings shortcodes * add processors * add worker installation shortcodes * add log source config shortcodes * remove old splunck hec doc * add archive log shortcodes * add sds and datadog archives * update prereqs * update shortcodes * add upgrade div and legacy warning shortcodes * add processors shortcode * splunk tcp docs * fix links * really fix all shortcode links * update all splunk tcp docs * add archives dest tcp * add sumo logic docs * fix sumo logic docs * add datadog agent docs * make titles consistent * consistency edits * add dd overviews * add cloudformation table * fix landing page links * add path notation example * small fix * add splt logs prereqs tabs * add gov note * small edits * small edits * small edits * Apply suggestions from code review Co-authored-by: Jen Gilbert <[email protected]> * Apply suggestions from code review Co-authored-by: Jen Gilbert <[email protected]> * capitalize placeholders and small edits * Update content/en/observability_pipelines/_index.md Co-authored-by: Jen Gilbert <[email protected]> * [DOCS-7432] Configure existing pipelines for OP (#22205) * add doc * small edits * add source env vars shortcode and update doc * update overview * update shortcode content * update shortcodes * small edit * fix link * Apply suggestions from code review Co-authored-by: Jen Gilbert <[email protected]> * Update layouts/shortcodes/observability_pipelines/configure_existing_pipelines/destination_env_vars/sumo_logic.md Co-authored-by: Jen Gilbert <[email protected]> * fix amazon eks shortcode * uppercase placeholders * small edits and fixes --------- Co-authored-by: Jen Gilbert <[email protected]> * remove sources list and add further reading to landing * add split logs images * add alt text and general proc image * try to fix image * try again to fix * desperately try to fix images * add images to md * add images * update images * remap to edit fields * remove datadog tab from destination env var * make us1-fed note red * cloudformation contact support * update config existing pipelines title and add links to that doc * update nav and architecture nav, title, and filename * Update content/en/observability_pipelines/log_volume_control/_index.md --------- Co-authored-by: Jen Gilbert <[email protected]> * add marketecture image * fix best practices filename and nav link * small edits * add aliases for monitoring, architecture, and troubleshooting legacy docs * Correct prettier issue * Fix indent --------- Co-authored-by: StefonSimmons <[email protected]> Co-authored-by: eugene kirillov <[email protected]> Co-authored-by: Esther Kim <[email protected]> Co-authored-by: Brian Deutsch <[email protected]> Co-authored-by: Jen Gilbert <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do? What is the motivation?
Merged into base branch's PR: #21974
Merge instructions
Additional notes