Stream conversation logs in sandbox #2073

sobolk · 2024-10-02T21:07:47Z

Problem

Conversation handler logs should show up when executing npx ampx sandbox --stream-function-logs.

Changes

This PR makes the following changes to stream and improve logs in conversation handler.

Adds logs with log level in conversation handler runtime componenets.
1. Conversation handler lambda is configured to emit structured logs. So that it supports log levels, see https://docs.aws.amazon.com/lambda/latest/dg/nodejs-logging.html for reference. I.e. we set log format to json as opposed to text (which doesn't support application log levels).
2. We add more info and debug logs around incoming event, bedrock invocations, tools invocation, emitting response. Debug logs dump request/response payloads.
We change Sandbox component to support arbitrary log group attached to lambda function.
1. Previously Sandbox assumed that log group is /aws/lambda/<functionName>. This however won't work for conversation handler (and later if we expose knobs related to logging configuration.
2. Conversation handler requires custom log group. I attempted to use <functionName> to mimic default convention, but the problem is that log group is input to function ctor, therefore it's impossible to use function's properties to create log group (as this happens first). The only alternative I found was to generate function name ourselves, but that wasn't great either - components used by function ctor internally are not available to us, so if we generate name it would look differently than other functions. Hence I figured that it's better to just make sandbox support arbitrary log group.
3. The ListTags call in Sandbox is replaced by GetFunction which gives both information that we need - tags and logGroupName. The AmplifyBackendDeployFullAccess already has necessary permissions to call GetFunction.
I moved outputs generation from backend-ai to ai-construct. So that data construct can pass outputStrategy to it in default scenario.

Out of scope

Setting log level for function at runtime. This should be covered separately with support configuring function log settings #2058 . Meanwhile customers can adjust the level on AWS Console after deployment.
This PR exposes ability to pass outputStrategy to ai construct. The necessary plumbing on data side will be pursued separately. The idea was validated by a temporary change (reverted here)

Validation

Added/modified unit tests.
Manual check

Manual check with default info level:

Manual check with log level set to debug via AWS Console:

Checklist

If this PR includes a functional change to the runtime behavior of the code, I have added or updated automated test coverage for this change.
If this PR requires a change to the Project Architecture README, I have included that update in this PR.
If this PR requires a docs update, I have linked to that docs PR above.
If this PR modifies E2E tests, makes changes to resource provisioning, or makes SDK calls, I have run the PR checks with the run-e2e label set.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

changeset-bot · 2024-10-02T21:07:51Z

🦋 Changeset detected

Latest commit: 88544ca

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 5 packages

Name	Type
@aws-amplify/ai-constructs	Minor
@aws-amplify/backend-ai	Minor
@aws-amplify/backend-output-schemas	Minor
@aws-amplify/backend	Patch
@aws-amplify/sandbox	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

sobolk · 2024-10-03T15:01:39Z

packages/ai-constructs/API.md

@@ -49,6 +51,7 @@ type ConversationHandlerFunctionProps = {
        modelId: string;
        region?: string;
    }>;
+    outputStorageStrategy?: BackendOutputStorageStrategy<FunctionOutput>;


This is similar to what data and auth construct do.

amplify-backend/packages/auth-construct/src/types.ts

Lines 472 to 475 in 27202b4

/**

* @internal

*/

outputStorageStrategy?: BackendOutputStorageStrategy<AuthOutput>;

sobolk · 2024-10-03T15:02:54Z

packages/sandbox/src/lambda_function_log_streamer.ts

+        new GetFunctionCommand({
+          FunctionName: functionName,


We already do have permissions to call GetFunction in AmplifyBackendDeployFullAccess

The reason we went with ListTags earlier was for performance reasons, since it will be called after each sandbox deployment.

Was ListTags significantly faster than GetFunction ?

Some considerations:

The number of requests to lambda CP doesn't change. We were calling ListTags per each function.

We eliminated extra CloudFormation calls to resolve ARNs.

The alternative would be to call GetFunctionConfiguration in addition to ListTags. But, making two requests is not great quota and latency wise.

It seems that default quota for GetFunction is much more generous than other CP APIs. See picture below.

Synced offline. We'll keep GetFunction given that alternatives are not better.

Amplifiyer · 2024-10-03T16:19:54Z

packages/ai-constructs/src/conversation/conversation_handler_construct.ts

+    outputStorageStrategy?.appendToBackendOutputList(functionOutputKey, {
+      version: '1',
+      payload: {
+        definedFunctions: this.resources.lambda.functionName,


But this is not a definedFunction. We should store this under it's own AIConstructOutputKey.

Amplifiyer · 2024-10-03T16:34:19Z

packages/ai-constructs/src/conversation/conversation_handler_construct.ts

@@ -67,6 +86,7 @@ export class ConversationHandlerFunction
          // For custom entry we do bundle SDK as we can't control version customer is coding against.
          bundleAwsSDK: !!this.props.entry,
        },
+        loggingFormat: LoggingFormat.JSON,


To be able to actually see the debug logs, don't we need to also expose the ApplicationLogLevel property so customers can configure it? if I recall cdk sets the default as INFO?

Correct. I called this out in the "Out of scope" section in the PR description.

packages/sandbox/src/lambda_function_log_streamer.ts

Co-authored-by: Amplifiyer <[email protected]>

This reverts commit 3e8b590.

This reverts commit 3c2d417.

sobolk · 2024-10-03T19:52:32Z

packages/backend-output-schemas/src/index.ts

+/**
+ * Expected key that AI conversation output is stored under
+ */
+export const aiConversationOutputKey = 'AWS::Amplify::AI::Conversation';


the naming here follows ai-constructs/<sub_usecase>
I.e. if we add another AI feature not related to conversation we'd have AWS::Amplify::AI::SomethingNew

…logging-improvements

Stream conversation logs in sandbox

9e385e8

sobolk added the run-e2e Label that will include e2e tests in PR checks workflow label Oct 2, 2024

sobolk added 7 commits October 2, 2024 14:16

fix that

12a8b88

refactor that

98e4d16

and that

bb7af36

this works

7bacac4

Undo the hack

3e8b590

more tests

6b01e25

that

9e57482

sobolk marked this pull request as ready for review October 3, 2024 14:59

sobolk requested review from a team as code owners October 3, 2024 14:59

sobolk commented Oct 3, 2024

View reviewed changes

Amplifiyer reviewed Oct 3, 2024

View reviewed changes

sobolk and others added 6 commits October 3, 2024 09:52

Update packages/sandbox/src/lambda_function_log_streamer.ts

abbb022

Co-authored-by: Amplifiyer <[email protected]>

use different output

00cd3ae

Revert "Undo the hack"

3c2d417

This reverts commit 3e8b590.

use different outputs

0a68474

use different outputs

7151725

Revert "Revert "Undo the hack""

12aa57b

This reverts commit 3c2d417.

sobolk commented Oct 3, 2024

View reviewed changes

sobolk added 2 commits October 3, 2024 12:54

Merge remote-tracking branch 'origin/main' into conversation-handler-…

abfae89

…logging-improvements

fix that

88544ca

Amplifiyer approved these changes Oct 4, 2024

View reviewed changes

rtpascual approved these changes Oct 4, 2024

View reviewed changes

sobolk merged commit 0a5e51c into main Oct 4, 2024
57 of 58 checks passed

sobolk deleted the conversation-handler-logging-improvements branch October 4, 2024 17:05

This was referenced Nov 12, 2024

fix(conversation): plumb outputStorageStrategy through transformer to handler aws-amplify/amplify-category-api#3014

Closed

fix: plumb outputStorageStrategy through conversation transformer to handler aws-amplify/amplify-category-api#3017

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stream conversation logs in sandbox #2073

Stream conversation logs in sandbox #2073

sobolk commented Oct 2, 2024 •

edited

Loading

changeset-bot bot commented Oct 2, 2024 •

edited

Loading

sobolk Oct 3, 2024

sobolk Oct 3, 2024

Amplifiyer Oct 3, 2024

sobolk Oct 3, 2024

sobolk Oct 3, 2024

Amplifiyer Oct 3, 2024

Amplifiyer Oct 3, 2024

sobolk Oct 3, 2024

sobolk Oct 3, 2024

	/**
	* @internal
	*/
	outputStorageStrategy?: BackendOutputStorageStrategy<AuthOutput>;

Stream conversation logs in sandbox #2073

Stream conversation logs in sandbox #2073

Conversation

sobolk commented Oct 2, 2024 • edited Loading

Problem

Changes

Out of scope

Validation

Checklist

changeset-bot bot commented Oct 2, 2024 • edited Loading

🦋 Changeset detected

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sobolk commented Oct 2, 2024 •

edited

Loading

changeset-bot bot commented Oct 2, 2024 •

edited

Loading