Confluent Kafka generator

This template will generate python scripts to create kafka topology in an existing Kafka cluster based on your AsyncApi.

The following Confluent Kafka components can be defined using AsyncApi specification:

• topics
• schema definitions
• [TODO] connectors
• [TODO] KSql

Table of Contents

• Usage
• Run it
• Special Considerations
  • Servers
  • Security
    • Environment variables
• AsyncApi Extensions
• Contributors

Usage

  Usage: ag [options] <asyncapi.yaml> @ekozynin/asyncapi-kafka-template

  Options:
    --force-write               force writing of the generated files to given directory (defaults to false)
    -o, --output <outputDir>    directory where to put the generated files (defaults to current directory)

Run it

The generated scripts were tested with python 3.9.7

Go to the root folder of the generated code and run this command (you need python):

  Usage: python main.py -e <environment>

  Options:
    environment   one of the "servers" definitions in your asyncapi file

First time you may need to install additional python dependencies. Go to the root folder of the generated code and install required python dependencies:

pip install -U -r python-requirements.txt

Special Considerations

Servers

Kafka cluster usually has different components to deploy Kafka topology to: brokers, schema registry, connectors, KSql.

To name the servers follow this rules:

• server name should start with the environment name (such as local, dev, prod)
• followed by dash '-'
• followed by component type. One of: 'broker', 'schemaRegistry'

Security

To connect to Confluent Cloud Cluster, the servers above should have 'security' attribute defined. Security schemas must be defined exactly as below. Refer to full example.

components:
  securitySchemes:
    confluentBroker:
      type: userPassword
      x-configs:
        security.protocol: sasl_ssl
        sasl.mechanisms: PLAIN
        sasl.username: '{{ CLUSTER_API_KEY }}'
        sasl.password: '{{ CLUSTER_API_SECRET }}'

    confluentSchemaRegistry:
      type: userPassword
      x-configs:
        basic.auth.user.info: '{{ SCHEMA_REGISTRY_API_KEY }}:{{ SCHEMA_REGISTRY_API_SECRET }}'

Environment variables

Use environment variables to pass security details when deploying kafka topology.

The names for environment variables are:

• CLUSTER_API_KEY
• CLUSTER_API_SECRET
• SCHEMA_REGISTRY_API_KEY
• SCHEMA_REGISTRY_API_SECRET

AsyncApi Extensions

Additional extensions to the AsyncApi specification that this generator understands.

Contributors

Eugen Kozynin 💻 📖 🎨 🤔 🚧

Jake Bayer 💻 👀 🤔