Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(documentation): updated folder structure #274

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 101 additions & 0 deletions dotnet-test-samples/async-architectures-lambda-sqs/Readme.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
[![.NET: 6.0](https://img.shields.io/badge/.NET-6.0-Green)]()
[![AWS: Lambda](https://img.shields.io/badge/AWS-Lambda-blueviolet)]()
[![Amazon: S3](https://img.shields.io/badge/Amazon-S3-blueviolet)]()
[![Amazon: SQS](https://img.shields.io/badge/Amazon-SQS-blueviolet)]()
[![test: unit](https://img.shields.io/badge/Test-Unit-blue)]()
[![test: integration](https://img.shields.io/badge/Test-Integration-yellow)]()

# Test Asynchronous Architectures

Automated tests are crucial components of event driven architectures. They help to ensure that your systems are developed efficiently, accurately and with high quality. This section provides some guidance for designing automated tests for event driven architectures and asynchronous systems using Python.

## View a Generic Asynchronous Pattern
Asynchronous systems typically receive messages or events and immediately store them for future processing. Later, a secondary processing service may perform operations on the stored data. Processed data may then be sent as output to additional services or placed into another storage location. Below is a diagram illustrating a generic asynchronous pattern.

![Generic Asynchronous System](./img/generic.png)

## Establish Logical Boundaries
Asynchronous systems rarely exist in isolation. Typically a production system will be made up of many interconnected subsystems. In order to create a reasonable testing strategy, it is useful to break complex systems into a set of logical subsystems. A subsystem may be a group of services that work together to accomplish a single task. A subsystem should have well understood inputs and outputs. A subsystem should be small enough to be able to reason about and understand. Breaking your complex architecture into smaller subsystems makes it easier to create isolated and targeted tests.

## Create Test Harnesses
When testing asynchronous systems, it can be useful to create test harnesses. These harnesses contain resources that generate inputs for your subsystem and then receive the system’s outputs. Your tests will use the harnesses to exercise the system and determine whether it is performing as expected. These harnesses are resources that are used for testing purposes only. They are not used by production features. Test harnesses are typically deployed to pre-production environments only. However, deploying test harnesses to production environments may be useful for production systems that are tolerant of test data. In these cases you may decide to perform tests in production.

## Configure Event Producers and Event Listeners
Test harnesses are usually composed of event producers and event listeners. The producers provide input to the System Under Test (SUT) and the listeners are configured to receive output. The automated tests send configured events to the producer and then poll the listener to examine the output. If the output meets expectations the test passes.

![Generic Asynchronous System Test](./img/generic-async-test.png)

## Define Service Level Agreements
Although your architecture may be asynchronous, it is still useful to establish reasonable expectations about the maximum duration your system may take to process before it considered to be in a failure state. These expectations may be explicitly defined as Service Level Agreements (SLAs). When you design your tests, you may set timeouts that match your SLA’s. If the system does not return results within the timeout period it can be considered to be in violation of the SLA and the tests will fail.


# Asynchronous Integration Test with Amazon Simple Queue Service (SQS)
You may use a variety of resource types to create the event listener for your asynchronous system under test ([more about event listeners](https://github.com/aws-samples/serverless-test-samples/tree/main/dotnet-test-samples/async-architectures#create-event-listeners-in-test-environments)).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the end of the refactoring this folder would be removed, need to find a solution for the common artifacts and pages.
Perhaps keeping the old page is a good idea but needs to update https://github.com/aws-samples/serverless-test-samples/tree/main/dotnet-test-samples/async-architectures to point at this pattern as well


In this pattern polling is used to wait for a specific message using long polling and busy loop.
## Review the System Under Test
The System Under Test (SUT) in this example is an asynchronous notification service. It contains a source S3 bucket that receives a file. A Lambda function is configured to be notified when the putObject event is executed on this bucket. the Lambda function then would queue a new notification message with the file's details in a SQS queue.
![S3 to Lambda to SQS](./img/s3-lambda-sqs.png)

Your goal is to test this asynchronous process. Since the end result of this workflow is a message in SQS you do not need any additional resources.
You will deploy the following resources:

* S3 Source Bucket
* Lambda function
* SQS Destination queue

This starter project consists of:
* serverless.template - an AWS CloudFormation Serverless Application Model template file for declaring your Serverless functions and other AWS resources
* Function.cs - class file containing a class with a single function handler method
* aws-lambda-tools-defaults.json - default argument settings for use with Visual Studio and command line deployment tools for AWS

You may also have a test project depending on the options selected.

The generated function handler responds to events on an Amazon S3 bucket. The handler receives the bucket and object key details in an S3Event instance and returns the content type of the object as the function output. Replace the body of this method, and parameters, to suit your needs.

After deploying your function you must configure an Amazon S3 bucket as an event source to trigger your Lambda function.
## Prerequisites
The SAM CLI is an extension of the AWS CLI that adds functionality for building and testing serverless applications. It contains features for building your appcation locally, deploying it to AWS, and emulating AWS services locally to support automated unit tests.

To use the SAM CLI, you need the following tools.

- SAM CLI - [Install the SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html)
- .NET 6 - [Install .NET 6](https://dotnet.microsoft.com/en-us/download)

## Build the project

```bash
dotnet build
```

## Deploy project resources to the cloud

```bash
sam build
sam deploy --guided
```

## Run the tests
There are three types of tests in this solution:
* S3Notification.UnitTests - unit tests that runs on local machine and do not need any environment setup
* S3Notification.IntegrationTests - require AWS account, use the aws cli to configure region and certifications
* S3Notification.E2ETests - system/end to end tests: before running tests you will need to run after deployment f the lambda function, ensure you set the two environment variables to the Stack Name and AWS Region used when deploying your resources to the cloud.

### Windows
```shell
$env:AWS_SAM_STACK_NAME = ""
$env:AWS_SAM_REGION_NAME = ""
dotnet test tests\S3Notification.E2ETests\S3Notification.E2ETests.csproj
```

### Linux
```bash
export AWS_SAM_STACK_NAME=""
export AWS_SAM_REGION_NAME=""
dotnet test tests/S3Notification.E2ETests/S3Notification.E2ETests.csproj
```

## Cleanup
```bash
sam delete
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio Version 17
VisualStudioVersion = 17.6.33815.320
MinimumVisualStudioVersion = 10.0.40219.1
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{521DD4B5-B08C-4AFC-8D93-3E5552DE2C2D}"
ProjectSection(SolutionItems) = preProject
src\template.yaml = src\template.yaml
Readme.md = Readme.md
EndProjectSection
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "src", "src", "{5E136BA6-1662-401B-8D0D-0BA0595A942D}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "tests", "tests", "{81458A7E-41B1-4553-AE5D-227E4E9A7D47}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "S3Notifications", "src\S3Notifications\S3Notifications.csproj", "{0645487F-46F6-4BA7-8334-2997E9CC70B9}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "S3Notifications.E2ETests", "tests\S3Notifications.E2ETests\S3Notifications.E2ETests.csproj", "{AE0182DD-A9F0-419D-89F5-F346AF920CD5}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "S3Notifications.UnitTests", "tests\S3Notifications.UnitTests\S3Notifications.UnitTests.csproj", "{C3905C57-8269-41C1-AB11-B5BA39577C66}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "S3Notifications.TestUtilities", "tests\S3Notifications.TestUtilities\S3Notifications.TestUtilities.csproj", "{33A91439-4C1F-4F97-9BC2-F3003008F010}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "S3Notifications.integrationTests", "tests\S3Notifications.integrationTests\S3Notifications.integrationTests.csproj", "{5FF22FB1-FE5D-45A9-98DE-8B5492106CCB}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Release|Any CPU = Release|Any CPU
EndGlobalSection
GlobalSection(ProjectConfigurationPlatforms) = postSolution
{0645487F-46F6-4BA7-8334-2997E9CC70B9}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{0645487F-46F6-4BA7-8334-2997E9CC70B9}.Debug|Any CPU.Build.0 = Debug|Any CPU
{0645487F-46F6-4BA7-8334-2997E9CC70B9}.Release|Any CPU.ActiveCfg = Release|Any CPU
{0645487F-46F6-4BA7-8334-2997E9CC70B9}.Release|Any CPU.Build.0 = Release|Any CPU
{AE0182DD-A9F0-419D-89F5-F346AF920CD5}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{AE0182DD-A9F0-419D-89F5-F346AF920CD5}.Debug|Any CPU.Build.0 = Debug|Any CPU
{AE0182DD-A9F0-419D-89F5-F346AF920CD5}.Release|Any CPU.ActiveCfg = Release|Any CPU
{AE0182DD-A9F0-419D-89F5-F346AF920CD5}.Release|Any CPU.Build.0 = Release|Any CPU
{C3905C57-8269-41C1-AB11-B5BA39577C66}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{C3905C57-8269-41C1-AB11-B5BA39577C66}.Debug|Any CPU.Build.0 = Debug|Any CPU
{C3905C57-8269-41C1-AB11-B5BA39577C66}.Release|Any CPU.ActiveCfg = Release|Any CPU
{C3905C57-8269-41C1-AB11-B5BA39577C66}.Release|Any CPU.Build.0 = Release|Any CPU
{33A91439-4C1F-4F97-9BC2-F3003008F010}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{33A91439-4C1F-4F97-9BC2-F3003008F010}.Debug|Any CPU.Build.0 = Debug|Any CPU
{33A91439-4C1F-4F97-9BC2-F3003008F010}.Release|Any CPU.ActiveCfg = Release|Any CPU
{33A91439-4C1F-4F97-9BC2-F3003008F010}.Release|Any CPU.Build.0 = Release|Any CPU
{5FF22FB1-FE5D-45A9-98DE-8B5492106CCB}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{5FF22FB1-FE5D-45A9-98DE-8B5492106CCB}.Debug|Any CPU.Build.0 = Debug|Any CPU
{5FF22FB1-FE5D-45A9-98DE-8B5492106CCB}.Release|Any CPU.ActiveCfg = Release|Any CPU
{5FF22FB1-FE5D-45A9-98DE-8B5492106CCB}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
EndGlobalSection
GlobalSection(NestedProjects) = preSolution
{0645487F-46F6-4BA7-8334-2997E9CC70B9} = {5E136BA6-1662-401B-8D0D-0BA0595A942D}
{AE0182DD-A9F0-419D-89F5-F346AF920CD5} = {81458A7E-41B1-4553-AE5D-227E4E9A7D47}
{C3905C57-8269-41C1-AB11-B5BA39577C66} = {81458A7E-41B1-4553-AE5D-227E4E9A7D47}
{33A91439-4C1F-4F97-9BC2-F3003008F010} = {81458A7E-41B1-4553-AE5D-227E4E9A7D47}
{5FF22FB1-FE5D-45A9-98DE-8B5492106CCB} = {81458A7E-41B1-4553-AE5D-227E4E9A7D47}
EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution
SolutionGuid = {6BE69204-644C-49EC-AE6A-E08CA582876F}
EndGlobalSection
EndGlobal
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are thos used in other async patterns? maybe move them to a common location such as https://github.com/aws-samples/serverless-test-samples/tree/main/dotnet-test-samples/docs

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{
"title": "Asynchronous Tests with Lambda and SQS",
"description": "This project shows how to test an asynchronous serverless workflow that triggers a lambda with SQS as backend service",
"content_language": "English",
"language": ".NET",
"type": ["Unit", "Integration", "End to end"],
"diagram": "../img/s3-lambda-sqs.png",
"git_repo_url": "https://github.com/aws-samples/serverless-test-samples",
"pattern_source": "AWS",
"framework": "SAM",
"services": ["s3", "lambda", "sqs"],
"pattern_detail_tabs": [
{
"title": "Application Code Example (GetStock Functions)",
"filepath": "/src/S3Notification/Functions.cs"
},
{
"title": "Unit Tests",
"filepath": "/tests/S3Notification.UnitTests/FunctionsTests.cs"
},
{
"title": "Integration Tests",
"filepath": "/tests/S3Notification.IntegrationTests/Usings.cs"
},
{
"title": "End to End Tests",
"filepath": "/tests/S3Notification.E2ETests/Usings.cs"
}
],
"authors": [
{
"name": "Dror Helper",
"image": "https://media.licdn.com/dms/image/C4D03AQGbBwfXtx0Zmg/profile-displayphoto-shrink_200_200/0/1638988504464?e=1686182400&v=beta&t=5G7912InrYUbzHXPah-84eIDnh1j1UYQKIRFEENKmBs",
"bio": "Senior Microsoft Specialist Solution Architect at AWS",
"linkedin": "https://www.linkedin.com/in/drorhelper/",
"twitter": "https://twitter.com/dhelper"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
using System.Text.Json;
using Amazon.Lambda.Core;
using Amazon.Lambda.S3Events;
using Amazon.SQS;
using Amazon.SQS.Model;

// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace S3Notifications;

public class Function
{
private const string QueueUrlEnvKey = "QUEUE_URL";
private IAmazonSQS SqsClient { get; }

/// <summary>
/// Default constructor. This constructor is used by Lambda to construct the instance. When invoked in a Lambda environment
/// the AWS credentials will come from the IAM role associated with the function and the AWS region will be set to the
/// region the Lambda function is executed in.
/// </summary>
public Function()
{
SqsClient = new AmazonSQSClient();
}

/// <summary>
/// Constructs an instance with a preconfigured SQS client. This can be used for testing the outside of the Lambda environment.
/// </summary>

public Function(IAmazonSQS sqsClient)
{
SqsClient = sqsClient;
}

/// <summary>
/// This method is called for every Lambda invocation. This method takes in an S3 event object and can be used
/// to respond to S3 notifications.
/// </summary>
/// <param name="evnt"></param>
/// <param name="context"></param>
/// <returns></returns>
public async Task<string?> FunctionHandler(S3Event evnt, ILambdaContext context)
{
if (evnt.Records?.Count is null or 0)
{
return null;
}

var s3Event = evnt.Records[0];

var message = new S3NotificationMessage(s3Event.S3.Bucket.Name, s3Event.S3.Object.Key, s3Event.EventName);

var queueName = Environment.GetEnvironmentVariable(QueueUrlEnvKey);

if (queueName == null)
{
throw new ApplicationException($"{QueueUrlEnvKey} was not set");
}

return await SendMessageToQueue(queueName, message);
}

private async Task<string> SendMessageToQueue(string queueUrl, S3NotificationMessage message)
{
var jsonString = JsonSerializer.Serialize(message);

var sendMessageRequest = new SendMessageRequest(queueUrl, jsonString);
var sendMessageResponse = await SqsClient.SendMessageAsync(sendMessageRequest);

return sendMessageResponse.MessageId;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
namespace S3Notifications;

public record S3NotificationMessage(string BucketName, string ObjectKey, string EventName);
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net6.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
<AWSProjectType>Lambda</AWSProjectType>
<!-- This property makes the build directory similar to a publish directory and helps the AWS .NET Lambda Mock Test Tool find project dependencies. -->
<CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
<!-- Generate ready to run images during publishing to improve cold start time. -->
<PublishReadyToRun>true</PublishReadyToRun>
</PropertyGroup>
<ItemGroup>
<None Remove=".DS_Store" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Amazon.Lambda.Core" Version="2.1.0" />
<PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="2.3.1" />
<PackageReference Include="Amazon.Lambda.S3Events" Version="3.0.1" />
<PackageReference Include="AWS.Lambda.Powertools.Logging" Version="1.1.1" />
<PackageReference Include="AWSSDK.S3" Version="3.7.200.1" />
<PackageReference Include="AWSSDK.SQS" Version="3.7.200.1" />
</ItemGroup>
</Project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"Information": [
"This file provides default values for the deployment wizard inside Visual Studio and the AWS Lambda commands added to the .NET Core CLI.",
"To learn more about the Lambda commands with the .NET Core CLI execute the following command at the command line in the project root directory.",
"dotnet lambda help",
"All the command line options for the Lambda command can be specified in this file."
],
"profile": "dhelper@isenguard",
"region": "us-east-1",
"configuration": "Release",
"s3-prefix": "async-lambda-sqs/",
"template": "serverless.template",
"template-parameters": ""
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Description: Template that creates a S3 bucket, SQS Queue and a Lambda function
that will be invoked when new objects are upload to the bucket.
Then queue a new message with that file details
Globals:
Function:
MemorySize: 256
Architectures: [arm64]
Runtime: dotnet6
Timeout: 300
Tracing: Active
Parameters:
BucketName:
Type: String
Description: Name of S3 bucket to be created.
The Lambda function will be invoked when new objects are upload to the bucket.
If left blank a name will be generated.
MinLength: "0"
Conditions:
BucketNameGenerated:
!Equals [!Ref BucketName, ""]
Resources:
S3Function:
Type: AWS::Serverless::Function
Properties:
CodeUri: ./S3Notifications
Handler: S3Notifications::S3Notifications.Function::FunctionHandler
Policies:
- AWSLambda_FullAccess
- Version: "2012-10-17"
Statement:
- Effect: Allow
Action: sqs:SendMessage
Resource: !GetAtt NotificationQueue.Arn
Environment:
Variables:
QUEUE_URL: !Ref NotificationQueue
Events:
NewS3Event:
Type: S3
Properties:
Bucket: !Ref MonitoredBucket
Events:
- s3:ObjectCreated:*
NotificationQueue:
Type: AWS::SQS::Queue
MonitoredBucket:
Type: AWS::S3::Bucket
Properties:
BucketName:
!If [BucketNameGenerated, !Ref AWS::NoValue, !Ref BucketName]
Outputs:
S3BucketName:
Value: !Ref MonitoredBucket
Description: Bucket that will invoke the lambda function when new objects are created.
QueueUrl:
Value: !Ref NotificationQueue
Description: Queue that would receive notification after S3 related events
Loading
Loading