Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(v2): batch validation with partial failure #1621

Open
wants to merge 15 commits into
base: v2
Choose a base branch
from
Open

feat(v2): batch validation with partial failure #1621

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
d22ad0c
validation of sqs / kinesis batches with partial failures
jeromevdl Apr 5, 2024
0923826
cleanup jdk8 stuff + slf4j simple logs for tests
jeromevdl Apr 5, 2024
9f05464
fix sonar potential NPE
jeromevdl Apr 8, 2024
af05320
merge v2 in branch
jeromevdl Jun 21, 2024
2240d80
documentation
jeromevdl Jun 24, 2024
e36b59a
Merge branch 'v2' into feat/validation-batch
jeromevdl Jun 24, 2024
0b86612
Merge branch 'v2' into feat/validation-batch
jeromevdl Jun 24, 2024
dfb5f0a
Merge branch 'v2' into feat/validation-batch
jeromevdl Jun 24, 2024
b830c83
update mockito (and bytebuddy) for java21 compat
jeromevdl Jul 1, 2024
5764fe5
Merge branch 'v2' into feat/validation-batch
jeromevdl Jul 1, 2024
5331bae
comment
jeromevdl Jul 2, 2024
0dcb8b9
Merge branch 'v2' into feat/validation-batch
jeromevdl Jul 2, 2024
503170a
Merge branch 'v2' into feat/validation-batch
jeromevdl Jul 2, 2024
92dec0e
Merge branch 'v2' into feat/validation-batch
jeromevdl Aug 2, 2024
d8afbba
fix mvn pom
jeromevdl Aug 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
122 changes: 30 additions & 92 deletions docs/utilities/validation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,12 @@ This utility provides JSON Schema validation for payloads held within events and
**Key features**

* Validate incoming events and responses
* Built-in validation for most common events (API Gateway, SNS, SQS, ...)
* Built-in validation for most common events (API Gateway, SNS, SQS, ...) and support for partial batch failures (SQS, Kinesis)
* JMESPath support validate only a sub part of the event

## Install
Depending on your version of Java (either Java 1.8 or 11+), the configuration slightly changes.

=== "Maven Java 11+"
=== "Maven"
```xml hl_lines="3-7 16 18 24-27"
<dependencies>
...
Expand Down Expand Up @@ -58,52 +57,7 @@ Depending on your version of Java (either Java 1.8 or 11+), the configuration sl
</build>
```

=== "Maven Java 1.8"

```xml hl_lines="3-7 16 18 24-27"
<dependencies>
...
<dependency>
<groupId>software.amazon.lambda</groupId>
<artifactId>powertools-validation</artifactId>
<version>{{ powertools.version }}</version>
</dependency>
...
</dependencies>
...
<!-- configure the aspectj-maven-plugin to compile-time weave (CTW) the aws-lambda-powertools-java aspects into your project -->
<build>
<plugins>
...
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>aspectj-maven-plugin</artifactId>
<version>1.14.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<complianceLevel>1.8</complianceLevel>
<aspectLibraries>
<aspectLibrary>
<groupId>software.amazon.lambda</groupId>
<artifactId>powertools-validation</artifactId>
</aspectLibrary>
</aspectLibraries>
</configuration>
<executions>
<execution>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
...
</plugins>
</build>
```

=== "Gradle Java 11+"
=== "Gradle"

```groovy hl_lines="3 11"
plugins {
Expand All @@ -123,27 +77,6 @@ Depending on your version of Java (either Java 1.8 or 11+), the configuration sl
targetCompatibility = 11 // or higher
```

=== "Gradle Java 1.8"

```groovy hl_lines="3 11"
plugins {
id 'java'
id 'io.freefair.aspectj.post-compile-weaving' version '6.6.3'
}

repositories {
mavenCentral()
}

dependencies {
aspect 'software.amazon.lambda:powertools-validation:{{ powertools.version }}'
}

sourceCompatibility = 1.8
targetCompatibility = 1.8
```


## Validating events

You can validate inbound and outbound events using `@Validation` annotation.
Expand All @@ -157,10 +90,15 @@ We support JSON schema version 4, 6, 7 and 201909 (from [jmespath-jackson librar
`@Validation` annotation is used to validate either inbound events or functions' response.

It will fail fast if an event or response doesn't conform with given JSON Schema. For most type of events a `ValidationException` will be thrown.

For API gateway events associated with REST APIs and HTTP APIs - `APIGatewayProxyRequestEvent` and `APIGatewayV2HTTPEvent` - the `@Validation`
annotation will build and return a custom 400 / "Bad Request" response, with a body containing the validation errors. This saves you from having
to catch the validation exception and map it back to a meaningful user error yourself.

For SQS and Kinesis events - `SQSEvent` and `KinesisEvent`- the `@Validation` annotation will add the invalid messages
jeromevdl marked this conversation as resolved.
Show resolved Hide resolved
to the batch item failures list in the response, respectively `SQSBatchResponse` and `StreamsEventResponse`
and removed from the event so that you do not process them within the handler.

While it is easier to specify a json schema file in the classpath (using the notation `"classpath:/path/to/schema.json"`), you can also provide a JSON String containing the schema.

=== "MyFunctionHandler.java"
Expand Down Expand Up @@ -217,31 +155,31 @@ For the following events and responses, the Validator will automatically perform

** Events **

Type of event | Class | Path to content |
------------------------------------------------- | ------------------------------------------------- | -------------------------------------------------
API Gateway REST | APIGatewayProxyRequestEvent | `body`
API Gateway HTTP | APIGatewayV2HTTPEvent | `body`
Application Load Balancer | ApplicationLoadBalancerRequestEvent | `body`
Cloudformation Custom Resource | CloudFormationCustomResourceEvent | `resourceProperties`
CloudWatch Logs | CloudWatchLogsEvent | `awslogs.powertools_base64_gzip(data)`
EventBridge / Cloudwatch | ScheduledEvent | `detail`
Kafka | KafkaEvent | `records[*][*].value`
Kinesis | KinesisEvent | `Records[*].kinesis.powertools_base64(data)`
Kinesis Firehose | KinesisFirehoseEvent | `Records[*].powertools_base64(data)`
Kinesis Analytics from Firehose | KinesisAnalyticsFirehoseInputPreprocessingEvent | `Records[*].powertools_base64(data)`
Kinesis Analytics from Streams | KinesisAnalyticsStreamsInputPreprocessingEvent | `Records[*].powertools_base64(data)`
SNS | SNSEvent | `Records[*].Sns.Message`
SQS | SQSEvent | `Records[*].body`
| Type of event | Class | Path to content |
|---------------------------------|-------------------------------------------------|----------------------------------------------|
| API Gateway REST | APIGatewayProxyRequestEvent | `body` |
| API Gateway HTTP | APIGatewayV2HTTPEvent | `body` |
| Application Load Balancer | ApplicationLoadBalancerRequestEvent | `body` |
| Cloudformation Custom Resource | CloudFormationCustomResourceEvent | `resourceProperties` |
| CloudWatch Logs | CloudWatchLogsEvent | `awslogs.powertools_base64_gzip(data)` |
| EventBridge / Cloudwatch | ScheduledEvent | `detail` |
| Kafka | KafkaEvent | `records[*][*].value` |
| Kinesis | KinesisEvent | `Records[*].kinesis.powertools_base64(data)` |
| Kinesis Firehose | KinesisFirehoseEvent | `Records[*].powertools_base64(data)` |
| Kinesis Analytics from Firehose | KinesisAnalyticsFirehoseInputPreprocessingEvent | `Records[*].powertools_base64(data)` |
| Kinesis Analytics from Streams | KinesisAnalyticsStreamsInputPreprocessingEvent | `Records[*].powertools_base64(data)` |
| SNS | SNSEvent | `Records[*].Sns.Message` |
| SQS | SQSEvent | `Records[*].body` |

** Responses **

Type of response | Class | Path to content (envelope)
------------------------------------------------- | ------------------------------------------------- | -------------------------------------------------
API Gateway REST | APIGatewayProxyResponseEvent} | `body`
API Gateway HTTP | APIGatewayV2HTTPResponse} | `body`
API Gateway WebSocket | APIGatewayV2WebSocketResponse} | `body`
Load Balancer | ApplicationLoadBalancerResponseEvent} | `body`
Kinesis Analytics | KinesisAnalyticsInputPreprocessingResponse} | `Records[*].powertools_base64(data)``
| Type of response | Class | Path to content (envelope) |
|-----------------------|---------------------------------------------|---------------------------------------|
| API Gateway REST | APIGatewayProxyResponseEvent} | `body` |
| API Gateway HTTP | APIGatewayV2HTTPResponse} | `body` |
| API Gateway WebSocket | APIGatewayV2WebSocketResponse} | `body` |
| Load Balancer | ApplicationLoadBalancerResponseEvent} | `body` |
| Kinesis Analytics | KinesisAnalyticsInputPreprocessingResponse} | `Records[*].powertools_base64(data)`` |

## Custom events and responses

Expand Down
149 changes: 52 additions & 97 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,8 @@
</modules>

<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
<maven.deploy.plugin.version>3.1.2</maven.deploy.plugin.version>
<log4j.version>2.20.0</log4j.version>
<log4j.version>2.23.1</log4j.version>
Expand All @@ -91,6 +91,7 @@
<aws-embedded-metrics.version>4.1.2</aws-embedded-metrics.version>
<jmespath.version>0.6.0</jmespath.version>
<elastic.version>1.6.0</elastic.version>
<mockito.version>5.6.0</mockito.version>

<!-- As we have a .mvn directory at the root of the project, this will evaluate to the root directory
regardless of where maven is run - sub-module, or root. -->
Expand Down Expand Up @@ -307,6 +308,12 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>${slf4j.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.skyscreamer</groupId>
<artifactId>jsonassert</artifactId>
Expand All @@ -318,6 +325,12 @@
<artifactId>aspectjtools</artifactId>
<version>${aspectj.version}</version>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>${mockito.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-tests</artifactId>
Expand Down Expand Up @@ -456,6 +469,43 @@
<autoReleaseAfterClose>true</autoReleaseAfterClose>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<systemPropertyVariables>
<!-- TODO: remove when updating mockito / bytebuddy with Java21 compat -->
jeromevdl marked this conversation as resolved.
Show resolved Hide resolved
<net.bytebuddy.experimental>true</net.bytebuddy.experimental>
</systemPropertyVariables>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<version>3.3.0</version>
<configuration>
<propertyExpansion>basedir=${project.rootdir}</propertyExpansion>
<configLocation>checkstyle.xml</configLocation>
<consoleOutput>true</consoleOutput>
<failsOnError>true</failsOnError>
<linkXRef>false</linkXRef>
</configuration>
<!-- does not work without this dependency -->
<dependencies>
<dependency>
<groupId>com.puppycrawl.tools</groupId>
<artifactId>checkstyle</artifactId>
<version>10.12.3</version>
</dependency>
</dependencies>
<executions>
<execution>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

Expand Down Expand Up @@ -585,101 +635,6 @@
</plugins>
</build>
</profile>
<profile>
<id>olderThanJdk11</id>
<activation>
<jdk>(,11)</jdk>
</activation>
<properties>
<!-- mockito 5+ is not compatible anymore with java < 11 -->
<mockito.version>4.11.0</mockito.version>
</properties>
<dependencies>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>${mockito.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-inline</artifactId>
<version>${mockito.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
</profile>
<profile>
<id>newerThanJdk11</id>
<activation>
<jdk>[11,)</jdk>
</activation>
<properties>
<mockito.version>5.6.0</mockito.version>
</properties>
<dependencies>
<!-- since mockito 5.3, no need to have mockito-inline -->
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>${mockito.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<systemPropertyVariables>
<!-- TODO: remove when updating mockito / bytebuddy with Java21 compat -->
<net.bytebuddy.experimental>true</net.bytebuddy.experimental>
</systemPropertyVariables>
</configuration>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>newerThanJdk8</id>
<activation>
<jdk>[9,)</jdk>
</activation>
<build>
<plugins>
<plugin>
<!-- we run checkstyle only on Java 11, no need to run it for all JDK, it's just code style -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<version>3.3.0</version>
<configuration>
<propertyExpansion>basedir=${project.rootdir}</propertyExpansion>
<configLocation>checkstyle.xml</configLocation>
<consoleOutput>true</consoleOutput>
<failsOnError>true</failsOnError>
<linkXRef>false</linkXRef>
</configuration>
<!-- does not work without this dependency -->
<!-- does not work with this dependency on Java 8 -->
<dependencies>
<dependency>
<groupId>com.puppycrawl.tools</groupId>
<artifactId>checkstyle</artifactId>
<version>10.12.3</version>
</dependency>
</dependencies>
<executions>
<execution>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>

</project>
10 changes: 10 additions & 0 deletions powertools-batch/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,16 @@
<artifactId>aws-lambda-java-tests</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

</project>
10 changes: 10 additions & 0 deletions powertools-cloudformation/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,16 @@
<artifactId>junit-jupiter-params</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
Expand Down
Loading
Loading