diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index 1c96b4353ebe2..08a7bee520d2f 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -2857,16 +2857,26 @@ menu: parent: serverless_step_functions identifier: serverless_step_functions_installation weight: 201 + - name: Merge Step Functions and Lambda Traces + url: serverless/step_functions/merge-step-functions-lambda + parent: serverless_step_functions + identifier: serverless_step_functions_merge_lambda_traces + weight: 202 - name: Enhanced Metrics url: serverless/step_functions/enhanced-metrics parent: serverless_step_functions identifier: serverless_step_functions_enhanced_metrics - weight: 202 + weight: 203 + - name: Redrive Executions + url: serverless/step_functions/redrive + parent: serverless_step_functions + identifier: serverless_step_functions_redrive + weight: 204 - name: Troubleshooting url: serverless/step_functions/troubleshooting parent: serverless_step_functions identifier: serverless_step_functions_troubleshooting - weight: 203 + weight: 205 - name: AWS Fargate identifier: serverless_aws_fargate url: integrations/ecs_fargate diff --git a/content/en/serverless/step_functions/_index.md b/content/en/serverless/step_functions/_index.md index 7138818c188aa..9efe1b15bd382 100644 --- a/content/en/serverless/step_functions/_index.md +++ b/content/en/serverless/step_functions/_index.md @@ -4,15 +4,18 @@ further_reading: - link: "/serverless/step_functions/installation" tag: "Documentation" text: "Install Serverless Monitoring for AWS Step Functions" -- link: "/serverless/step_functions/troubleshooting" +- link: "/serverless/step_functions/merge-step-functions-lambda" tag: "Documentation" - text: "Troubleshoot Serverless Monitoring for AWS Step Functions" -- link: "/integrations/amazon_step_functions/" + text: "Merge Step Functions and Lambda Traces" +- link: "/serverless/step_functions/enhanced-metrics" tag: "Documentation" - text: "AWS Step Functions integration" -- link: "/serverless/aws_lambda/" + text: "Enhanced Metrics for AWS Step Functions" +- link: "/serverless/step_functions/redrive" tag: "Documentation" - text: "Serverless Monitoring for AWS Lambda" + text: "Redrive Executions" +- link: "/serverless/step_functions/troubleshooting" + tag: "Documentation" + text: "Troubleshoot Serverless Monitoring for AWS Step Functions" --- AWS Step Functions is a serverless orchestration service that lets developers create and manage multi-step application workflows. In addition to getting Cloudwatch metrics from Datadog's [AWS Step Functions integration][2], Datadog also provides AWS Step Function tracing, logs, and [enhanced metrics][3] through the collection of Cloudwatch logs. diff --git a/content/en/serverless/step_functions/installation.md b/content/en/serverless/step_functions/installation.md index 6985e36ddeb9c..077a21d6fd533 100644 --- a/content/en/serverless/step_functions/installation.md +++ b/content/en/serverless/step_functions/installation.md @@ -11,7 +11,6 @@ further_reading: ### Requirements * The full Step Function execution length must be less than 6 hours for full traces. -* Linked Lambda traces are supported for Node.js (layer v112+) and Python (layer v95+) runtimes. ### How it works AWS Step Functions is a fully managed service, and the Datadog Agent cannot be directly installed on Step Functions. However, Datadog can monitor Step Functions through Cloudwatch metrics and logs. @@ -25,7 +24,7 @@ Datadog uses these ingested logs to generate [enhanced metrics][8] and traces fo {{< img src="serverless/step_functions/telemetry_ingestion.png" alt="A diagram explaining how Step Functions telemetry is ingested and used in Datadog" style="width:100%;" >}} -### Setup +## Setup Ensure that the [AWS Step Functions integration][9] is installed. @@ -55,17 +54,13 @@ For developers using [Serverless Framework][4] to deploy serverless applications apiKeySecretArn: forwarderArn: enableStepFunctionsTracing: true - propagateUpstreamTrace : true ``` - Replace `` with {{< region-param key="dd_site" code="true" >}} (ensure the correct SITE is selected on the right). - Replace `` with the ARN of the AWS secret where your [Datadog API key][3] is securely stored. The key needs to be stored as a plaintext string (not a JSON blob). The `secretsmanager:GetSecretValue` permission is required. For quick testing, you can instead use `apiKey` and set the Datadog API key in plaintext. - Replace `` with the ARN of your Datadog Lambda Forwarder, as noted previously. - - `propagateUpstreamTrace`: Optional. Set to `true` to inject Step Function context into downstream Lambda and Step Function invocations For additional settings, see [Datadog Serverless Framework Plugin - Configuration parameters][7]. -4. For Node.js and Python runtimes, set `mergeStepFunctionAndLambdaTraces:true` in your `serverless.yaml` file. This links your Step Function traces with Lambda traces. If you have not instrumented your Lambda functions to send traces, you can [follow the steps to add the Lambda layer for your preferred runtime][8]. - [1]: https://docs.datadoghq.com/serverless/libraries_integrations/plugin/ [2]: /logs/guide/forwarder/ [3]: https://app.datadoghq.com/organization-settings/api-keys @@ -73,7 +68,6 @@ For developers using [Serverless Framework][4] to deploy serverless applications [5]: /logs/guide/forwarder/?tab=cloudformation#upgrade-to-a-new-version [6]: logs/guide/forwarder/?tab=cloudformation#installation [7]: https://github.com/datadog/serverless-plugin-datadog?tab=readme-ov-file#configuration-parameters -[8]: /serverless/installation/#installation-instructions {{% /tab %}} {{% tab "Datadog CLI" %}} 1. If you have not already, install the [Datadog CLI][1] v2.18.0+. @@ -90,25 +84,20 @@ For developers using [Serverless Framework][4] to deploy serverless applications datadog-ci stepfunctions instrument \ --step-function \ --forwarder \ - --env \ - --propagate-upstream-trace - + --env ``` - Replace `` with the ARN of your Step Function. Repeat the `--step-function` flag for each Step Function you wish to instrument. - Replace `` with the ARN of your Datadog Lambda Forwarder, as noted previously. - Replace `` with the environment tag you would like to apply to your Step Functions. - - `--propagate-upstream-trace` is optional, and updates your Step Function definitions to inject Step Function context into any downstream Step Function or Lambda invocations. For more information about the `datadog-ci stepfunctions` command, see the [Datadog CLI documentation][5]. -4. For Node.js and Python runtimes, add the flag `--merge-step-function-and-lambda-traces` in your datadog-ci command. This links your Step Function traces with Lambda traces. If you have not yet instrumented your Lambda functions to send traces, you can [follow the steps to add the Lambda layer for your preferred runtime][6]. [1]: /serverless/libraries_integrations/cli/ [2]: /logs/guide/forwarder/ [3]: /logs/guide/forwarder/?tab=cloudformation#upgrade-to-a-new-version [4]: logs/guide/forwarder/?tab=cloudformation#installation [5]: https://github.com/DataDog/datadog-ci/blob/master/src/commands/stepfunctions/README.md -[6]: /serverless/installation/#installation-instructions {{% /tab %}} {{% tab "Custom" %}} @@ -136,86 +125,10 @@ For developers using [Serverless Framework][4] to deploy serverless applications 4. Under *Log group*, select the log group for your state machine. For example, `/aws/vendedlogs/states/my-state-machine`. 5. Enter a filter name. You can choose to name it "empty filter" and leave the *Filter pattern* box blank. -
If you are using a different instrumentation method such as Serverless Framework or datadog-ci, enabling autosubscription may create duplicated logs. Choose one configuration method to avoid this behavior.
+
If you are using a different instrumentation method, such as Serverless Framework or datadog-ci, enabling autosubscription may create duplicated logs. To avoid this behavior, choose only one configuration method.
4. Set up tags. Open your AWS console and go to your Step Functions state machine. Open the *Tags* section and add `env:`, `service:`, and `version:` tags. The `env` tag is required to see traces in Datadog, and it defaults to `dev`. The `service` tag defaults to the state machine's name. The `version` tag defaults to `1.0`. -5. Link your Step Function traces to downstream Lambda traces or nested Step Function traces: - -{{% collapse-content title="Link Step Function traces to downstream Lambda traces" level="h4" %}} -For Node.js and Python runtimes, you can link your Step Function traces to Lambda traces. On the Lambda Task, set the `Parameters` key with the following: - -```json -"Parameters": { - "Payload.$": "States.JsonMerge($$, $, false)", - ... -} -``` - -The `JsonMerge` [intrinsic function][1] merges the [Step Functions context object][2] (`$$`) with the original Lambda's input payload (`$`). Fields of the original payload overwrite the Step Functions context object if their keys are the same. - -**Example**: - -{{< highlight json "hl_lines=5-5" >}} -"Lambda Read From DynamoDB": { - "Type": "Task", - "Resource": "arn:aws:states:::lambda:invoke", - "Parameters": { - "Payload.$": "States.JsonMerge($$, $, false)", - "FunctionName": "${lambdaArn}" - }, - "End": true -} -{{< /highlight >}} - -Alternatively, if you have business logic defined in the payload, you could also use the following: - -{{< highlight json "hl_lines=7-9" >}} -"Lambda Read From DynamoDB": { - "Type": "Task", - "Resource": "arn:aws:states:::lambda:invoke", - "Parameters": { - "Payload": { - ... - "Execution.$": "$$.Execution", - "State.$": "$$.State", - "StateMachine.$": "$$.StateMachine" - }, - "FunctionName": "${lambdaArn}" - }, - "End": true -} -{{< /highlight >}} - -If you have not yet instrumented your Lambda functions to send traces, you can [follow the steps to add the Lambda layer for your preferred runtime][3]. - -[1]: https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-intrinsic-functions.html#asl-intrsc-func-json-manipulate -[2]: https://docs.aws.amazon.com/step-functions/latest/dg/input-output-contextobject.html -[3]: /logs/guide/forwarder/?tab=cloudformation#installation -{{% /collapse-content %}} - -{{% collapse-content title="Link Step Function traces to nested Step Function traces" level="h4" %}} -**Example**: - -{{< highlight json "hl_lines=9-13" >}} -"Step Functions StartExecution": { - "Type": "Task", - "Resource": "arn:aws:states:::states:startExecution", - "Parameters": { - "StateMachineArn": "${stateMachineArn}", - "Input": { - "StatePayload": "Hello from Step Functions!", - "AWS_STEP_FUNCTIONS_STARTED_BY_EXECUTION_ID.$": "$$.Execution.Id", - "CONTEXT": { - "Execution.$": "$$.Execution", - "State.$": "$$.State", - "StateMachine.$": "$$.StateMachine" - } - } - }, - "End": true -} -{{< /highlight >}} -{{% /collapse-content %}} + [1]: /logs/guide/forwarder/ [2]: /logs/guide/forwarder/?tab=cloudformation#upgrade-to-a-new-version @@ -224,29 +137,31 @@ If you have not yet instrumented your Lambda functions to send traces, you can [ {{% /tab %}} {{< /tabs >}} -## Enable enhanced metrics - -Datadog generates [enhanced metrics][8] from collected Cloudwatch logs. To enable this, add a `DD_ENHANCED_METRICS` tag to each of your Step Functions and set the value to `true`. - -Enhanced metrics are automatically enabled if you enable traces. - ## Enable tracing +
To see Step Functions spans and Lambda spans connected in the same trace, see Merge Step Functions with your AWS Lambda traces.
+ Datadog generates traces from collected Cloudwatch logs. To enable this, add a `DD_TRACE_ENABLED` tag to each of your Step Functions and set the value to `true`. Alternatively, to enable tracing for **all** your Step Functions, add a `DD_STEP_FUNCTIONS_TRACE_ENABLED` environment variable to the Datadog Forwarder and set the value to `true`. Enhanced metrics are automatically enabled if you enable tracing.
If you enable enhanced metrics without enabling traces, you are only billed for Serverless Workload Monitoring. If you enable tracing (which automatically includes enhanced metrics), you are billed for both Serverless Workload Monitoring and Serverless APM. See Pricing.
+## Merge Step Functions with your AWS Lambda traces + +See [Merge Step Functions traces with Lambda traces][11]. Ensure that you have also [set up Serverless Monitoring for AWS Lambda][10]. + ## Sample traces To manage the APM traced invocation sampling rate for serverless functions, set the `DD_TRACE_SAMPLE_RATE` environment variable on the function to a value between 0.000 (no tracing of Step Function invocations) and 1.000 (trace all Step Function invocations). The dropped traces are not ingested into Datadog. -## Link Step Functions with your AWS Lambda traces +## Enable enhanced metrics (without tracing) + +Datadog generates [enhanced metrics][8] from collected Cloudwatch logs. Enhanced metrics are automatically enabled if you enable traces. -Ensure that you have also [set up Serverless Monitoring for AWS Lambda][10]. +To enable enhanced metrics without enabling tracing, add a `DD_ENHANCED_METRICS` tag to each of your Step Functions and set the value to `true`. ## See your Step Function metrics, logs, and traces in Datadog @@ -256,16 +171,6 @@ After you have invoked your state machine, go to the [**Serverless app**][2] in If you cannot see your traces, see [Troubleshooting][5]. -### Retry Step Functions executions or redrive failed executions from within Datadog - -You can retry any execution that is being monitored or [redrive][11] executions to continue failed AWS Step Functions from the point of failure, without needing a complete state machine restart. This can be done directly from Datadog. - -To take action on you [Step Functions][12] in Datadog, you can either use the invocations list on a Step Function side-panel and click on the **Failed** pill to open a redrive modal, or open the Step Function Trace Map to retry an execution or redrive a failed execution. - -{{< img src="serverless/step_functions/redrive.png" alt="A visualization of a failed Step Function execution." style="width:100%;" >}} - -To enable using redrive within Datadog, configure an [AWS Connection][13] with [Datadog App Builder][14]. Ensure that your IAM roles include policies that have permissions to allow executing a Step Function for the retry action (`StartExecution`) or redriving a Step Function for the redrive action (`RedriveExecution`). - [2]: https://app.datadoghq.com/functions?search=&cloud=aws&entity_view=step_functions [3]: /serverless/installation/#installation-instructions [5]: /serverless/step_functions/troubleshooting @@ -274,7 +179,4 @@ To enable using redrive within Datadog, configure an [AWS Connection][13] with [ [8]: /serverless/step_functions/enhanced-metrics [9]: /integrations/amazon_step_functions [10]: /serverless/aws_lambda/installation -[11]: https://docs.aws.amazon.com/step-functions/latest/dg/redrive-executions.html -[12]: https://app.datadoghq.com/functions?cloud=aws&entity_view=step_functions -[13]: https://docs.aws.amazon.com/dtconsole/latest/userguide/welcome-connections.html -[14]: /service_management/app_builder/ +[11]: /serverless/step_functions/merge-step-functions-lambda diff --git a/content/en/serverless/step_functions/merge-step-functions-lambda.md b/content/en/serverless/step_functions/merge-step-functions-lambda.md new file mode 100644 index 0000000000000..36fee13e79cd7 --- /dev/null +++ b/content/en/serverless/step_functions/merge-step-functions-lambda.md @@ -0,0 +1,142 @@ +--- +title: Merge Step Functions and Lambda Traces +further_reading: + - link: '/serverless/step_functions/installation' + tag: 'Documentation' + text: 'Install Serverless Monitoring for AWS Step Functions' +--- + +This page describes how to merge your AWS Step Functions traces with related AWS Lambda traces or nested Step Functions traces. These instructions assume that you have already instrumented these [AWS Step Functions][1] and [Lambda functions][2] to send traces to Datadog. + +## Merge Step Functions traces with downstream Lambda traces + +### Requirements +Node.js (layer v112+) or Python (layer v95+) runtimes. + +### Setup + +{{< tabs >}} +{{% tab "Serverless Framework" %}} + +In your `serverless.yaml` file, set `mergeStepFunctionAndLambdaTraces` to `true`. For example: + +{{< highlight yaml "hl_lines=8" >}} +custom: + datadog: + site: + apiKeySecretArn: + forwarderArn: + enableStepFunctionsTracing: true + propagateUpstreamTrace: true + mergeStepFunctionAndLambdaTraces: true +{{< /highlight >}} + +{{% /tab %}} +{{% tab "Datadog CLI" %}} + +Run the following `datadog-ci` command: + +{{< highlight yaml "hl_lines=6" >}} +datadog-ci stepfunctions instrument \ + --step-function \ + --forwarder \ + --env \ + --propagate-upstream-trace \ + --merge-step-function-and-lambda-traces +{{< /highlight >}} + +The `merge-step-function-and-lambda-traces` flag lets you inject Step Functions context into downstream Lambda and Step Functions invocations. + +{{% /tab %}} +{{% tab "Custom" %}} + +On the Lambda Task, set the `Parameters` key as follows: + +```json +"Parameters": { + "Payload.$": "States.JsonMerge($$, $, false)", + ... +} +``` + +The `JsonMerge` [intrinsic function][1] merges the [Step Functions context object][2] (`$$`) with the original Lambda's input payload (`$`). Fields of the original payload overwrite the Step Functions context object if their keys are the same. + +**Example**: + +{{< highlight json "hl_lines=5-5" >}} +"Lambda Read From DynamoDB": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Parameters": { + "Payload.$": "States.JsonMerge($$, $, false)", + "FunctionName": "${lambdaArn}" + }, + "End": true +} +{{< /highlight >}} + +Alternatively, if you have business logic defined in the payload, you can also use the following format: + +{{< highlight json "hl_lines=7-9" >}} +"Lambda Read From DynamoDB": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Parameters": { + "Payload": { + ... + "Execution.$": "$$.Execution", + "State.$": "$$.State", + "StateMachine.$": "$$.StateMachine" + }, + "FunctionName": "${lambdaArn}" + }, + "End": true +} +{{< /highlight >}} + +[1]: https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-intrinsic-functions.html#asl-intrsc-func-json-manipulate +[2]: https://docs.aws.amazon.com/step-functions/latest/dg/input-output-contextobject.html + +{{% /tab %}} +{{< /tabs >}} + +## Merge upstream Lambda traces with Step Functions traces + +### Requirements +For Node.js: Datadog Lambda Library for Node.js layer v112+ **or** `dd-trace-js` v3.58.0, v4.37.0, v5.13.0. + +For Python: Datadog Lambda Library for Python layer 99+ **or** `dd-trace-py` v2.13.0. + +### Setup + +If the layer or tracer version requirements are fulfilled, no further setup is required. + +## Merge Step Functions traces with nested Step Functions traces + +To link your Step Function traces to nested Step Function traces, configure your task according to the following example: +{{< highlight json "hl_lines=9-13" >}} +"Step Functions StartExecution": { + "Type": "Task", + "Resource": "arn:aws:states:::states:startExecution", + "Parameters": { + "StateMachineArn": "${stateMachineArn}", + "Input": { + "StatePayload": "Hello from Step Functions!", + "AWS_STEP_FUNCTIONS_STARTED_BY_EXECUTION_ID.$": "$$.Execution.Id", + "CONTEXT": { + "Execution.$": "$$.Execution", + "State.$": "$$.State", + "StateMachine.$": "$$.StateMachine" + } + } + }, + "End": true +} +{{< /highlight >}} + +## Merge Lambda -> Step Functions -> Lambda + +This capability is not supported. + +[1]: /serverless/installation/#installation-instructions +[2]: /serverless/step_functions/installation \ No newline at end of file diff --git a/content/en/serverless/step_functions/redrive.md b/content/en/serverless/step_functions/redrive.md new file mode 100644 index 0000000000000..957a04f12f219 --- /dev/null +++ b/content/en/serverless/step_functions/redrive.md @@ -0,0 +1,34 @@ +--- +title: Redrive AWS Step Functions executions +further_reading: + - link: 'https://docs.aws.amazon.com/step-functions/latest/dg/redrive-executions.html' + tag: 'AWS Developer Guide' + text: 'Restarting state machine executions with redrive in Step Functions' + - link: "/service_management/app_builder/" + tag: "Documentation" + text: "Datadog App Builder" +--- + +This page explains how to [redrive][1] executions directly from Datadog to continue failed AWS Step Functions from the point of failure without a state machine restart. + +{{< img src="serverless/step_functions/redrive.png" alt="A visualization of a failed Step Function execution." style="width:100%;" >}} + +## Enable redrive within Datadog +To enable using redrive within Datadog, configure an [AWS Connection][3] with [Datadog App Builder][4]. Ensure that your IAM roles include permissions that allow executing a Step Function for the retry action (`StartExecution`) or redriving a Step Function for the redrive action (`RedriveExecution`). + +## Usage +To take action on a Step Function in Datadog: +1. Go to the [Step Functions][2] page. +2. Find the Step Function you wish to redrive. +3. Open this Step Function's side panel. On the **Executions** tab, locate the failed execution you wish to redrive. +4. Click on the **Failed** pill to open a redrive modal. +5. Click the **Redrive** button. + +## Further Reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: https://docs.aws.amazon.com/step-functions/latest/dg/redrive-executions.html +[2]: https://app.datadoghq.com/functions?cloud=aws&entity_view=step_functions +[3]: https://docs.aws.amazon.com/dtconsole/latest/userguide/welcome-connections.html +[4]: /service_management/app_builder/