Skip to content

What is Zerocode Testing

Jump to bottom
authorjapps edited this page May 8, 2020 · 52 revisions

Table of Contents

What Is Declarative Testing

Here is some similar insight from IEEE-

We propose a software testing paradigm called declarative testing. In declarative testing, a test scenario focuses on what to accomplish rather than on the imperative details of how to manipulate the state of an application under test and verify the final application state against an expected state. Declarative testing is a test design paradigm which separates test automation code into conceptual Answer, Executor, and Verifier entities.

In computer science, declarative programming is a programming paradigm—a style of building the structure and elements of computer programs—that expresses the logic of a computation without describing its control flow.

Advantages of Declarative Testing

  • According to IEEE -

Preliminary experience with declarative testing suggests that the modular characteristics of the paradigm may significantly enhance the ability of a testing effort to keep pace with the evolution of a software application during the application's development process.

  • Instead of writing code how to achieve the testing goals, we write what to achieve in the test intentions i.e. the test input and the expectations

  • Here the framework, behind the scene, handles the execution via necessary code to do the job for us e.g. API calls, DB executions, producing/consuming from Kafka topics etc

  • In this style we attempt to minimize or eliminate side effects by describing what the test must accomplish in terms of the business functionality, rather than describe how to accomplish it via programming or coding


That makes the test automation a lot easy and clean.


In the Declarative Style we don't need to write any of the below.

  • The Http Client (or Kafka Client) calls for REST APIs
  • Request payload parsing
  • Response payload parsing
  • Code for assertions/verifications e.g. comparing actual vs expected response
Declarative Style Traditional Style
"url":"/api/v1/register/persons" Create an HttpClient object. Set the url to "/api/v1/register/persons"
e.g. RequestBuilder.setUri(httpUrl);
"method": "POST" Set this POST operaton to the HttpClient object
e.g. RequestBuilder.create(methodName).setUri(httpUrl);
"request": { ... } Parse the request payload and set to HttpEntity.
e.g. HttpEntity httpEntity = EntityBuilder.create().setContentType(APPLICATION_JSON).setText(reqBody).build();
None. Nothing to do. Parse the response to Java object or JSON String
"verify": {JSON as-it-is} Compare the actual response against expected field by field.
- Use multiple assertThat(...).
- Traverse through the response Object field by field
- Or use JSON Path to extract value
Display all the mismatches and fail the test(time saver) Stop at first mismatch and fail the test(unwanted delay in getting feedback)
Straight forward and easy Step chaining is not straight forward

Drawing a Simile

To draw a simile, we can pay attention to how docker-compose works. In docker-compose we tell the Docker-Compose framework(in a YAML file) to spin up certain things at certain ports etc, and then, things are done for us by the framework.

That's declarative way of doing things

e.g. of a compose YAML file

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:5.0.1
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
  kafka:
    image: confluentinc/cp-kafka:5.0.1
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

How neat and compact is that? Just think of it, for instance, if we had to write code/shell-scripts for the same repetitive tasks, how much hassle we would have gone through!

Example of a Zerocode YAML Test Scenario(more>>) is below.

---
scenarioName: As simple GET request response
steps:
- name: "find_match"
  url: "/api/v1/search/persons"
  method: "GET"
  request:
    queryParams:
      lang: "Amazing"
      city: "Lon"
  verify:
    status: 200
    body:
      exactMatches: true
      name: "Mr Bean"

Testing Without Writing Code.

e.g.

YAML large

That's the declarative way of validating an API what we discussed earlier


Test Case Fields

  1. Http(REST API and SOAP)
  2. Kafka (Produce, Consume RAW vs JSON)
  3. Java Function call e.g. DB SQL Executror

Http(REST API and SOAP)

SCENARIO

Scenario means a Test-Scenario or an User-Journey or a Use-Case during test automation. It is represented in the following way.

"scenarioName": "Free text - Validate a POST and GET method for a customer"

LOOP

loop means the same Test-Scenario to be executed a number of times.

e.g.

    "scenarioName": "Free text - A scenario"
    "loop": 3

IGNORESTEPFAILURES

When this DSL flag is set to true, the framework will go ahead and execute the subsequent steps in the scenario file.

    "ignoreStepFailures": true

This is an optional flag and you can skip this or set to false to retain the default behavior.

URL

REST endpoint or a SOAP end-point or a Kafka topic.

    "url": "/api/v1/register/persons",

Or you can mention the FQDN with http or https with port

    "url": "https://apphost.gov.uk/api/v1/register/persons",

See ahead examples on how you can point to a Kafka topic using this url field.

METHOD

REST end-point or SOAP end-point All Http methods such as POST, PUT, GET, PATCH, DELETE etc

    "method": "POST",

Or when we need to call a Java function

    "method": "executeSql",

Or

when we need to validate Kafka events

    "operation": "produce",
or
    "operation": "consume",

Note- method and operation are identical and can be used interchangeably. Preferably method is used for http calls and operation is used for Kafka calls.

(See Kafka DSLs below)

RETRY

Retry comes handy when the actual response doesn't match the expected values in certain use-cases.

           "retry": {
                "max": 5,
                "delay": 2000
            },

The above settings will retry maximum of 5 times with 2sec delay between the retries.

If one fo the retries goes success(meaning if the actual response matches the expected response), then the framework will stop retrying further and come out of that step marking the it as PASSED.

CUSTOMLOG

Custom Log This is an optional field which can be used when user want custom log for particular step.

           "customLog": "custom message"

REQUEST

For REST end-point or SOAP end-point, request details with Headers and Body payload

           "request": {
                "body": {
                    "id": 1000,
                    "name": "Titan"
                }
            },

Or when we need to call a Java function with a SQL query as method parameter

    "request": "select id, name from customers"

QUERYPARAMS

This DSL field can be used for sending query params to the HTTP endpoints.

"queryParams":{
   "param1": "value1",
   "param2": "value2"
}

which is equivalent to ?param1=value1&param2=value2

HEADERS

Request with headers and body payload,

           "request": {
                "headers": {
                    "X-GOVT-TOKEN": "90945"
                },
                "body": {
                    "id": 1000,
                    "name": "Titan"
                }
            },

VERIFYMODE

  • verifyMode is STRICT or LENIENT
{
   "verifyMode": "STRICT",
   "verify":{
      ...
   }
}

When we specify STRICT mode, then the actual payload has to exactly match the expected payload.

LENIENT is the default mode even if we do not mention it.

VERIFY

Verifications and Assertions are used for the similar purpose where,

  • verify is mostly used for verification of an implementation against a Spec
  • assertions is mostly used for validation an implementation

For REST services, we need to put the expected response with response Status, Headers and Body payload.

Only status validation

           "verify": {
                "status": 200
            }

or

           "verify": {
                "status": 200
            }

Or status and payload id assertions Only status assertion

           "verify": {
                "status": 200,
                "body": {
                    "id" : 583231
                }
            }

Or partial or full payload assertions

            "verify": {
                "status": 200,
                "body": {
                    "login" : "octocat",
                    "id" : 583231,
                    "type" : "User"
                }
            }

Or with response headers details

           "verify": {
                "status": 200,
                "headers":{
                  "Server":"sit2.hsbc.co.uk",
                  "X-HSBC-BANK":"$NOT.NULL" //<--- "$NOT.NULL" if value is undeterministic
                },
                "body": {
                    "login" : "octocat",
                    "id" : 583231,
                    "type" : "User"
                }
            }

STATUS

For REST services or SOAP, we need to put the expected response with response Status, Headers and Body payload.

Only status assertion

           "verify": {
                "status": 200
            }

BODY

This is the payload, if present in the request section, then passed to the REST endpoint. This is the payload, if present in the response section, then treated as expected response from the REST endpoint.

           "verify": {
                "status": 200
                "body": {
                    "login" : "octocat",
                    "id" : 583231,
                    "type" : "User"
                }
            }

Kafka

TOPIC

We mention the Kafka topic name

    "url": "kafka-topic:heathrow-inbound",

OPERATION

We need to mention produce or consume from/to a Kafka topic

    "operation": "produce",

or

    "operation": "consume",

REQUEST

We need to Produce or Consume to/from a Kafka topic,

  • a RAW record
           "request": {
                "records": [
                    {
                        "key": "key-101",
                        "value": "Hello Kafka"
                    }
                ]
            },
  • a JSON record
           "request": {
                "recordType" : "JSON",
                "records": [
                    {
                        "key": "key-101",
                        "value": {
                            "name" : "Jey"
                        }
                    }
                ]
            },

Or while Consuming we can specify whether to commitSync after consuming, recordType as RAW or JSON etc.

            "request": {
                "consumerLocalConfigs": {
                    "recordType" : "JSON",
                    "commitSync": true,
                    "maxNoOfRetryPollsOrTimeouts": 3
                }
            },

ASSERTIONS

For Kafka services, we can put the expected response with response Status, RecordMetadata.

Only status assertion

           "verify": {
                "status": "Ok"
            }

Or status with recordMetadata assertion while Producing

           "verify": {
                "status": "Ok",
                "recordMetadata": "$NOT.NULL"
            }

Or size with records assertion while Consuming

           "verify": {
                "size": 1,
                "records": [
                    {
                        "key": 101,
                        "value": {
                            "name" : "Jey"
                        }
                    }

                ]
            }

HelloWorld Examples (Try at home)

More About Kafka, DB, OAuth2, Http etc (Click to expand)

The purpose of Zerocode lib is to make your API tests easy to write, easy to change, easy to share.

See the Table Of Contents for usages and examples.

For Kafka testing approach, visit this page Kafka-Testing Quick Start.

Running the Tests using JUnit

  • All examples above run via Junit @Test annotation like below.
@TargetEnv("github_host.properties")
@RunWith(ZeroCodeUnitRunner.class)
public class JustHelloWorldTest {

    @Test
    @Scenario("helloworld/hello_world_status_ok_assertions.json")
    public void testGet() throws Exception {

    }
}

You point to any JSON file and run. Hosts details are in the .properties file by @TargetEnv

  • Also you can run as a Suite pointing to the root of a package.

Both Declarative and Extensible

While Zerocode framework is light-weight and simple to write test intentions in JSON/YAML format, at the same time we can customize/extend it to add our own flavours.

For instance, we can add custom Http Headers to the entire test-suite or an individual test-case, automate OAuth2 secured APIs, or use our own flavour of Apache Kafka Client to deal with Kafka Brokers and much more stuff.

...making all these things super easy and straight forward.

Visit the Zerocode Documentation Site for all things.

Blogs

Clone this wiki locally