Event driven architectures decouple producers and consumers at the infrastructure layer, but these resources may still be coupled at the application layer by the event contract. Consumers may write business logic that expects events to conform to specific schemas. If schemas change over time the consumer code may fail. Creating automated tests that validate event schemas can help increase the quality of your systems and prevent breaking changes.
To prevent event schema changes from breaking consumer applications.
Schema tests are designed to check for changes that are not backward-compatible, such as removing elements, changing a field type or adding new mandatory elements. Adding optional elements is considered backward-compatible. If a new schema version is backward-compatible, the schema tests will pass and the consumer application business logic should not fail.
This test is integration service agnostic, as it runs against the schema published to a schema registry. Amazon EventBridge Schema Registry is one example of such a registry. It provides schemas for all events that are generated by AWS services. When you turn on schema discovery it will automatically detect and provide schemas for your custom events (OpenAPI and JSON Schemas). You can also use your own Git repository as a registry for storing schemas. In our test we will use a local “schemas” folder as a registry for simplicity.
The diagram below shows an example of how schema tests may run as part of a producer’s CI/CD pipeline. You can view a sample schema test that could be run during the "Build" step of the process.
Schema tests have two limitations worth mentioning.
First, schemas are inherently ambiguous as they do not capture business requirements. They may not express that a particular combination of fields is invalid, or they may not explicitly state the dependencies between fields. For example, when you create an AWS Lambda function, the runtime property is mandatory for zip packaging formats, but this field is not required for container image packaging formats. The schema for the Lambda create-function API payload will not convey this requirement. If an event consumer makes invalid assumptions about a new version of the payload, its business logic may fail even if there are technically no breaking schema changes.
Second, unless an event consumer uses SDK/code bindings generated specifically for the version of schema in use, the code and the schema may drift apart over time and the consumer business logic may eventually fail. In these cases, the schema validation test results create a false sense of security.
Both of these limitations can be addressed with contract testing.
To identify the side effects of non-breaking schema changes on consumer applications.
Contract testing uses sample events provided by producers to validate consumer side business logic and to provide a stronger guarantee of preventing breaking changes. Contract tests are owned by consumers.
Consider the following example. In a schema test, changing the format of an address field from a full address to a street address only and then adding new event fields like country, city, state and postcode would not produce a failure. These schema changes would be considered to be backward compatible, since the changes only add new elements. In contrast, a consumer test expecting a full address in the address field would fail and notify us that this change is actually not backward compatible.
There are other common strategies for implementing contract testing. One strategy is to establish test doubles. Test doubles are resources that replace production objects for testing purposes. The contract tests with test doubles run on the consumer side and expect that calls to the doubles return the same results as calls to the production application. Review the Pact.io documentation for examples of this approach.
This test is integration service agnostic as it uses sample events generated from a schema published to a schema registry. The diagram below shows one way of implementing contract testing as part of producer’s CI/CD pipeline.
The diagram below shows an example of how contract testing can be used as part of a producer’s CI/CD pipeline. You can view a sample contract test that could be run during the "Build" step of the process.
Writing tests and performing on-going maintenance requires a degree of additional effort. However once written, schema changes can be done more efficiently and with less effort to prevent and resolve breaking changes.
You may choose to be selective in deciding where to implement contract testing. Start by identifying critical applications where the cost of breaking changes is highest. Applications with greater tolerances for failure may choose to rely on schema testing only.
- SchemaTest.java - unit test for schema testing
- ContractTest.java - unit test for contract testing
- BaseJsonSchemaValidatorTest - Base class for json schema validator based on https://github.com/networknt/json-schema-validator
- CustomerValidator.java - Validator that contain business logic to check real value of properties
- schemas - versioned JSON Draft-07 schemas used for schema testing
- events - sample events used for contract testing
The event schema is documented using JSON Draft-07 specification. We will compare each of new schema's event against original/old schema using json schema validator. If no validation message exist in results, then it's working or backward compatible. If we pass new json events to original/old schema and found any validation message, then those new schema are broken or not backward compatible.
In order to detect breaking changes using contract testing, we will pass json event directly to business logic class CustomerValidator.java and validate using its public method directly.
Run with maven command test directly to test both schema and contract testing
mvn test