Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(v2): batch validation with partial failure #1621

Open
wants to merge 15 commits into
base: v2
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 28 additions & 23 deletions docs/utilities/validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This utility provides JSON Schema validation for payloads held within events and
**Key features**

* Validate incoming events and responses
* Built-in validation for most common events (API Gateway, SNS, SQS, ...)
* Built-in validation for most common events (API Gateway, SNS, SQS, ...) and support for partial batch failures (SQS, Kinesis)
* JMESPath support validate only a sub part of the event

## Install
Expand Down Expand Up @@ -92,10 +92,15 @@ The validator is configured to enable format assertions by default even for 2019
`@Validation` annotation is used to validate either inbound events or functions' response.

It will fail fast if an event or response doesn't conform with given JSON Schema. For most type of events a `ValidationException` will be thrown.

For API gateway events associated with REST APIs and HTTP APIs - `APIGatewayProxyRequestEvent` and `APIGatewayV2HTTPEvent` - the `@Validation`
annotation will build and return a custom 400 / "Bad Request" response, with a body containing the validation errors. This saves you from having
to catch the validation exception and map it back to a meaningful user error yourself.

For SQS and Kinesis events - `SQSEvent` and `KinesisEvent`- the `@Validation` annotation will add the invalid messages
jeromevdl marked this conversation as resolved.
Show resolved Hide resolved
to the batch item failures list in the response, respectively `SQSBatchResponse` and `StreamsEventResponse`
and removed from the event so that you do not process them within the handler.

While it is easier to specify a json schema file in the classpath (using the notation `"classpath:/path/to/schema.json"`), you can also provide a JSON String containing the schema.

=== "MyFunctionHandler.java"
Expand Down Expand Up @@ -152,31 +157,31 @@ For the following events and responses, the Validator will automatically perform

** Events **

Type of event | Class | Path to content |
------------------------------------------------- | ------------------------------------------------- | -------------------------------------------------
API Gateway REST | APIGatewayProxyRequestEvent | `body`
API Gateway HTTP | APIGatewayV2HTTPEvent | `body`
Application Load Balancer | ApplicationLoadBalancerRequestEvent | `body`
Cloudformation Custom Resource | CloudFormationCustomResourceEvent | `resourceProperties`
CloudWatch Logs | CloudWatchLogsEvent | `awslogs.powertools_base64_gzip(data)`
EventBridge / Cloudwatch | ScheduledEvent | `detail`
Kafka | KafkaEvent | `records[*][*].value`
Kinesis | KinesisEvent | `Records[*].kinesis.powertools_base64(data)`
Kinesis Firehose | KinesisFirehoseEvent | `Records[*].powertools_base64(data)`
Kinesis Analytics from Firehose | KinesisAnalyticsFirehoseInputPreprocessingEvent | `Records[*].powertools_base64(data)`
Kinesis Analytics from Streams | KinesisAnalyticsStreamsInputPreprocessingEvent | `Records[*].powertools_base64(data)`
SNS | SNSEvent | `Records[*].Sns.Message`
SQS | SQSEvent | `Records[*].body`
| Type of event | Class | Path to content |
|---------------------------------|-------------------------------------------------|----------------------------------------------|
| API Gateway REST | APIGatewayProxyRequestEvent | `body` |
| API Gateway HTTP | APIGatewayV2HTTPEvent | `body` |
| Application Load Balancer | ApplicationLoadBalancerRequestEvent | `body` |
| Cloudformation Custom Resource | CloudFormationCustomResourceEvent | `resourceProperties` |
| CloudWatch Logs | CloudWatchLogsEvent | `awslogs.powertools_base64_gzip(data)` |
| EventBridge / Cloudwatch | ScheduledEvent | `detail` |
| Kafka | KafkaEvent | `records[*][*].value` |
| Kinesis | KinesisEvent | `Records[*].kinesis.powertools_base64(data)` |
| Kinesis Firehose | KinesisFirehoseEvent | `Records[*].powertools_base64(data)` |
| Kinesis Analytics from Firehose | KinesisAnalyticsFirehoseInputPreprocessingEvent | `Records[*].powertools_base64(data)` |
| Kinesis Analytics from Streams | KinesisAnalyticsStreamsInputPreprocessingEvent | `Records[*].powertools_base64(data)` |
| SNS | SNSEvent | `Records[*].Sns.Message` |
| SQS | SQSEvent | `Records[*].body` |

** Responses **

Type of response | Class | Path to content (envelope)
------------------------------------------------- | ------------------------------------------------- | -------------------------------------------------
API Gateway REST | APIGatewayProxyResponseEvent} | `body`
API Gateway HTTP | APIGatewayV2HTTPResponse} | `body`
API Gateway WebSocket | APIGatewayV2WebSocketResponse} | `body`
Load Balancer | ApplicationLoadBalancerResponseEvent} | `body`
Kinesis Analytics | KinesisAnalyticsInputPreprocessingResponse} | `Records[*].powertools_base64(data)``
| Type of response | Class | Path to content (envelope) |
|-----------------------|---------------------------------------------|---------------------------------------|
| API Gateway REST | APIGatewayProxyResponseEvent} | `body` |
| API Gateway HTTP | APIGatewayV2HTTPResponse} | `body` |
| API Gateway WebSocket | APIGatewayV2WebSocketResponse} | `body` |
| Load Balancer | ApplicationLoadBalancerResponseEvent} | `body` |
| Kinesis Analytics | KinesisAnalyticsInputPreprocessingResponse} | `Records[*].powertools_base64(data)`` |

## Custom events and responses

Expand Down
134 changes: 40 additions & 94 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@
<aws-embedded-metrics.version>4.1.2</aws-embedded-metrics.version>
<jmespath.version>0.6.0</jmespath.version>
<elastic.version>1.6.0</elastic.version>
<mockito.version>5.12.0</mockito.version>

<!-- As we have a .mvn directory at the root of the project, this will evaluate to the root directory
regardless of where maven is run - sub-module, or root. -->
Expand Down Expand Up @@ -307,6 +308,12 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>${slf4j.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.skyscreamer</groupId>
<artifactId>jsonassert</artifactId>
Expand All @@ -318,6 +325,12 @@
<artifactId>aspectjtools</artifactId>
<version>${aspectj.version}</version>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>${mockito.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-tests</artifactId>
Expand Down Expand Up @@ -456,6 +469,33 @@
<autoReleaseAfterClose>true</autoReleaseAfterClose>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<version>3.3.0</version>
<configuration>
<propertyExpansion>basedir=${project.rootdir}</propertyExpansion>
<configLocation>checkstyle.xml</configLocation>
<consoleOutput>true</consoleOutput>
<failsOnError>true</failsOnError>
<linkXRef>false</linkXRef>
</configuration>
<!-- does not work without this dependency -->
<dependencies>
<dependency>
<groupId>com.puppycrawl.tools</groupId>
<artifactId>checkstyle</artifactId>
<version>10.12.3</version>
</dependency>
</dependencies>
<executions>
<execution>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

Expand Down Expand Up @@ -585,100 +625,6 @@
</plugins>
</build>
</profile>
<profile>
<id>olderThanJdk11</id>
<activation>
<jdk>(,11)</jdk>
</activation>
<properties>
<!-- mockito 5+ is not compatible anymore with java < 11 -->
<mockito.version>4.11.0</mockito.version>
</properties>
<dependencies>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>${mockito.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-inline</artifactId>
<version>${mockito.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
</profile>
<profile>
<id>newerThanJdk11</id>
<activation>
<jdk>[11,)</jdk>
</activation>
<properties>
<mockito.version>5.6.0</mockito.version>
</properties>
<dependencies>
<!-- since mockito 5.3, no need to have mockito-inline -->
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>${mockito.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<systemPropertyVariables>
<!-- TODO: remove when updating mockito / bytebuddy with Java21 compat -->
<net.bytebuddy.experimental>true</net.bytebuddy.experimental>
</systemPropertyVariables>
</configuration>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>newerThanJdk8</id>
<activation>
<jdk>[9,)</jdk>
</activation>
<build>
<plugins>
<plugin>
<!-- we run checkstyle only on Java 11, no need to run it for all JDK, it's just code style -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<version>3.3.0</version>
<configuration>
<propertyExpansion>basedir=${project.rootdir}</propertyExpansion>
<configLocation>checkstyle.xml</configLocation>
<consoleOutput>true</consoleOutput>
<failsOnError>true</failsOnError>
<linkXRef>false</linkXRef>
</configuration>
<!-- does not work without this dependency -->
<dependencies>
<dependency>
<groupId>com.puppycrawl.tools</groupId>
<artifactId>checkstyle</artifactId>
<version>10.12.3</version>
</dependency>
</dependencies>
<executions>
<execution>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>

</project>
10 changes: 10 additions & 0 deletions powertools-batch/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,16 @@
<artifactId>aws-lambda-java-tests</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

</project>
10 changes: 10 additions & 0 deletions powertools-cloudformation/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,16 @@
<artifactId>junit-jupiter-params</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
Expand Down
10 changes: 10 additions & 0 deletions powertools-common/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,16 @@
<artifactId>assertj-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
Expand Down
10 changes: 10 additions & 0 deletions powertools-idempotency/powertools-idempotency-core/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -38,5 +38,15 @@
<artifactId>powertools-serialization</artifactId>
</dependency>
<!-- Test dependencies -->
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
</project>
10 changes: 10 additions & 0 deletions powertools-large-messages/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,16 @@
<artifactId>junit-pioneer</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
Expand Down
10 changes: 10 additions & 0 deletions powertools-logging/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,16 @@
<artifactId>junit-jupiter-engine</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
Expand Down
5 changes: 5 additions & 0 deletions powertools-logging/powertools-logging-log4j/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,11 @@
<artifactId>junit-jupiter-engine</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
Expand Down
5 changes: 5 additions & 0 deletions powertools-logging/powertools-logging-logback/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,11 @@
<artifactId>junit-jupiter-engine</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
Expand Down
10 changes: 10 additions & 0 deletions powertools-metrics/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,16 @@
<artifactId>junit-jupiter-engine</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
Expand Down
Loading
Loading