From 130bd960f304294e098273c1ae942193884002a9 Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Tue, 27 Aug 2024 18:14:04 +0200 Subject: [PATCH 1/6] update: Kafka getting started --- docs/products/kafka/get-started.md | 227 ++++++++++++++++++++++++++++- 1 file changed, 219 insertions(+), 8 deletions(-) diff --git a/docs/products/kafka/get-started.md b/docs/products/kafka/get-started.md index a7b8f3526..d13493cbb 100644 --- a/docs/products/kafka/get-started.md +++ b/docs/products/kafka/get-started.md @@ -4,19 +4,230 @@ sidebar_label: Get started keywords: [quick start] --- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import ConsoleLabel from "@site/src/components/ConsoleIcons" import CreateService from "@site/static/includes/create-service-console.md" -Start using Aiven for Apache Kafka® by creating and configuring a service, connecting to it, and playing with sample data. +Start using Aiven for Apache Kafka® by creating and configuring a service, connecting to it, and working with your data. ## Create an Aiven for Apache Kafka® service - + + + + + + + + +1. [Create an authentication token](/docs/platform/howto/create_authentication_token). +1. Create a `sample.tf` file for the `aiven` provider configuration and the `aiven_kafka` + resource. + + ```hcl + variable "aiven_token" { + type = string + } + + variable "aiven_project_name" { + type = string + } + + terraform { + required_providers { + aiven = { + source = "aiven/aiven" + version = ">=4.0.0, <5.0.0" + } + } + } + + provider "aiven" { + api_token = var.aiven_token + } + + resource "aiven_kafka" "kafka" { + project = var.aiven_project_name + cloud_name = "google-europe-west1" + plan = "startup-16" + service_name = "my-kafka" + maintenance_window_dow = "friday" + maintenance_window_time = "23:00:00" + + kafka_user_config { + kafka_version = "2.7" + } + } + + output "kafka_service_host" { + value = aiven_kafka.kafka.service_host + } + + output "kafka_service_port" { + value = aiven_kafka.kafka.service_port + } + + output "kafka_service_username" { + value = aiven_kafka.kafka.service_username + } + + output "kafka_service_password" { + value = aiven_kafka.kafka.service_password + sensitive = true + } + ``` + +1. Create the `terraform.tfvars` file to assign values to your declared variables. + + ```hcl + aiven_token = "AIVEN_TOKEN" + aiven_project_name = "PROJECT_NAME" + ``` + +1. Run `terraform init` > `terraform plan` > `terraform apply --auto-approve`. +1. Store Terraform outputs in environment variables to use them for [connecting](#connect-to-service): + + ```bash + KAFKA_HOST="$(terraform output -raw kafka_service_host)" + KAFKA_PORT="$(terraform output -raw kafka_service_port)" + KAFKA_USER="$(terraform output -raw kafka_service_username)" + KAFKA_PASSWORD="$(terraform output -raw kafka_service_password)" + ``` + + + + +## Create an Apache Kafka® topic + + + + +1. Log in to the [Aiven Console](https://console.aiven.io/) and select the + Aiven for Apache Kafka® service. +1. Click in the sidebar. +1. Click **Create topic** and enter a name for the topic. +1. If required, set the advanced configuration option to **Yes**. +1. Set properties like replication factor, number of partitions, and other settings. + These can be changed later. +1. Click **Create topic**. + + + + + +1. Determine the topic specifications, including the number of partitions, replication + factor, and other settings. +1. Run the following command to create the `exampleTopic` topic: + + ```bash + avn service topic-create \ + --project demo-kafka-project \ + --service-name demo-kafka-service \ + --topic exampleTopic \ + --partitions 2 \ + --replication 2 + ``` + + Parameters: + - `avn service topic-create`: Creates a topic. + - `--project demo-kafka-project`: Specifies the project name. + - `--service-name demo-kafka-service`: Specifies the Aiven for Apache Kafka® service name. + - `--topic exampleTopic`: Specifies the name of the topic to create. + - `--partitions 2`: Specifies the number of partitions for the topic. + - `--replication 2`: Specifies the replication factor for the topic. + + + + +## Connect and produce/consume messages + +Connect to your Aiven for Apache Kafka service to start producing and consuming messages. +The **Quick connect** in the Aiven Console helps you establish a connection. + +1. Log in to the [Aiven Console](https://console.aiven.io/), select your project, and + the Aiven for Apache Kafka® service. +1. On the page of your service, click + **Quick connect**. +1. In the **Connect** window, select a tool or language to connect to your service, + follow the connection instructions, and click **Done**. + +After connecting, use your client to produce messages to your Apache Kafka® topics +and consume messages using the appropriate consumer setup. + +## Integration with Aiven for Apache Kafka + +Aiven for Apache Kafka® supports a wide range of integrations that extend its +capabilities, allowing you to seamlessly connect with external systems. Some key integration options include: + +- [Apache Kafka Connect](/docs/products/kafka/kafka-connect): Integrate + with external systems like databases and file systems. +- [Apache Kafka MirrorMaker](/docs/products/kafka/kafka-mirrormaker): + Enable cross-cluster replication. +- [Prometheus and Datadog](/docs/products/kafka/howto/kafka-prometheus-privatelink): + Monitor and manage Kafka metrics. +- [Elasticsearch](/docs/integrations/send-logs-to-elasticsearch) and OpenSearch: Store + logs and perform search and analytics. +- [Apache Flink® and Apache Kafka® Streams](/docs/products/kafka/howto/kafka-streams-with-aiven-for-kafka): + Process and analyze real-time data streams. + +These integrations can be configured within the Aiven Console +under **Integration endpoints**, allowing you to connect your Kafka service +with other platforms and tools efficiently. + +## View topic catalog + +The Aiven for Apache Kafka® topic catalog provides a centralized interface to view, +request, and manage topics across projects. It simplifies topic management and ensures +governance within your Aiven for Apache Kafka infrastructure. + +1. Log in to the [Aiven Console](https://console.aiven.io/). +1. Click **Tools**. +1. Select Apache Kafka topic catalog. + +Explore more in the [Aiven for Apache Kafka® topic catalog](/docs/products/kafka/concepts/topic-catalog-overview). + +## Enable governance + +Governance in Aiven for Apache Kafka® provides a secure and efficient way to manage your +Apache Kafka clusters. It centralizes control through policies and roles, ensuring +compliance and enforcing standards across your Aiven for Apache Kafka environments. + +:::note +You need super admin permissions to enable governance. +::: + +1. Access the [Aiven Console](https://console.aiven.io/) and click **Admin**. +1. On the organization page, click . +1. Click **Enable governance**. + +Manage topic creation and ownership requests for Apache Kafka® resources, ensuring +they meet governance standards. + +Discover more in the [Aiven for Apache Kafka® governance overview](/docs/products/kafka/concepts/governance-overview). + +## Configure Aiven for Apache Kafka® Connect + +Aiven for Apache Kafka® Connect lets you integrate your Aiven for Apache Kafka +service with other data sources and sinks. Set up Apache Kafka Connect in your project +to start building data pipelines. + +1. Ensure your project includes a Kafka service. +1. Create a dedicated Kafka Connect service via the Aiven Console. +1. Set up and deploy connectors to integrate with external systems. + +## Enable and use Karapace + +Karapace provides schema registry and REST APIs for managing schemas and interacting +with Aiven for Apache Kafka. + +1. Enable Karapace schema registry and REST APIs in your Kafka service. +2. Use Karapace to manage schemas and access Kafka via REST APIs. + +Learn more about [Karapace schema registry](/docs/products/kafka/karapace/concepts/schema-registry-authorization). + ## Next steps -- Check our [examples - project](https://github.com/aiven/aiven-examples) to find code - samples to get your application connected. -- Try our [sample data generator - project](https://github.com/aiven/python-fake-data-producer-for-apache-kafka) - to give you some data to get started with. +- Explore our [examples project](https://github.com/aiven/aiven-examples) for code samples. +- Use our [sample data generator project](https://github.com/aiven/python-fake-data-producer-for-apache-kafka) to create test data. From 897dd9196c6b33ba7b658af15b1e0f8798854be7 Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Wed, 28 Aug 2024 14:24:57 +0200 Subject: [PATCH 2/6] update: Kafka getting started --- docs/products/kafka/get-started.md | 178 ++++++++++++++++++++++------- 1 file changed, 139 insertions(+), 39 deletions(-) diff --git a/docs/products/kafka/get-started.md b/docs/products/kafka/get-started.md index d13493cbb..165798e61 100644 --- a/docs/products/kafka/get-started.md +++ b/docs/products/kafka/get-started.md @@ -9,10 +9,19 @@ import TabItem from '@theme/TabItem'; import ConsoleLabel from "@site/src/components/ConsoleIcons" import CreateService from "@site/static/includes/create-service-console.md" -Start using Aiven for Apache Kafka® by creating and configuring a service, connecting to it, and working with your data. +Start using Aiven for Apache Kafka® by setting up and configuring a service, connecting to it, and managing your data. + +## Prerequisites + +- Access to the [Aiven Console](https://console.aiven.io) +- [Aiven Provider for Terraform](https://registry.terraform.io/providers/aiven/aiven/latest/docs) +- [Aiven CLI](/docs/tools/cli) ## Create an Aiven for Apache Kafka® service +You can create your Kafka service using the Aiven Console, the Aiven CLI, or automate +the process with Terraform. + @@ -21,7 +30,7 @@ Start using Aiven for Apache Kafka® by creating and configuring a service, conn -1. [Create an authentication token](/docs/platform/howto/create_authentication_token). +1. [Create a personal token](/docs/platform/howto/create_authentication_token). 1. Create a `sample.tf` file for the `aiven` provider configuration and the `aiven_kafka` resource. @@ -78,15 +87,23 @@ Start using Aiven for Apache Kafka® by creating and configuring a service, conn } ``` -1. Create the `terraform.tfvars` file to assign values to your declared variables. +1. Create a `terraform.tfvars` file to assign values to your declared variables. ```hcl aiven_token = "AIVEN_TOKEN" aiven_project_name = "PROJECT_NAME" ``` -1. Run `terraform init` > `terraform plan` > `terraform apply --auto-approve`. -1. Store Terraform outputs in environment variables to use them for [connecting](#connect-to-service): +1. Run the following commands to initialize and apply the configuration: + + ```bash + terraform init + terraform plan + terraform apply --auto-approve + ``` + +1. Store the Terraform outputs in environment variables to use them for + [connecting](#connect-and-produceconsume-messages): ```bash KAFKA_HOST="$(terraform output -raw kafka_service_host)" @@ -95,6 +112,28 @@ Start using Aiven for Apache Kafka® by creating and configuring a service, conn KAFKA_PASSWORD="$(terraform output -raw kafka_service_password)" ``` + + + +Open your terminal and run the following command: + +```bash +avn service create SERVICE_NAME \ + --service-type kafka \ + --cloud CLOUD_REGION \ + --plan SERVICE_PLAN \ + -c kafka_connect=true \ + --disk-space-gib STORAGE_SIZE_GIB +``` + +Parameters: + +- `SERVICE_NAME`: The name for your Aiven for Apache Kafka service. +- `CLOUD_REGION`: The cloud region where the service is deployed. For example, + `google-europe-west3`. +- `SERVICE_PLAN`: The Aiven subscription plan. For example, `business-4`. +- `STORAGE_SIZE_GIB`: The total disk space for data storage in GiB. For example, `600`. + @@ -112,7 +151,6 @@ Start using Aiven for Apache Kafka® by creating and configuring a service, conn These can be changed later. 1. Click **Create topic**. - @@ -140,41 +178,123 @@ Start using Aiven for Apache Kafka® by creating and configuring a service, conn -## Connect and produce/consume messages +## Connect and produce/consume messages {#connect-and-produceconsume-messages} Connect to your Aiven for Apache Kafka service to start producing and consuming messages. The **Quick connect** in the Aiven Console helps you establish a connection. +### Connect to your Aiven for Apache Kafka service + 1. Log in to the [Aiven Console](https://console.aiven.io/), select your project, and the Aiven for Apache Kafka® service. 1. On the page of your service, click **Quick connect**. -1. In the **Connect** window, select a tool or language to connect to your service, - follow the connection instructions, and click **Done**. - -After connecting, use your client to produce messages to your Apache Kafka® topics -and consume messages using the appropriate consumer setup. +1. In the **Connect** window, choose your preferred tool or language, and follow the + connection instructions provided. +1. Select your authentication method: + - **Client certificate**: Use this method if your environment requires secure + communication using SSL/TLS certificates. For information on downloading CA + certificates, see [TLS/SSL certificates](https://aiven.io/docs/platform/concepts/tls-ssl-certificates#download-ca-certificates). + - **SASL**: Use this method if your environment uses Simple Authentication and + Security Layer (SASL) for authentication. +1. Follow the connection instructions provided based on the chosen method. +1. Click **Done**. + +### Produce and consume messages + +After connecting, use your Apache Kafka client to produce messages to your +Apache Kafka® topics and consume messages using the appropriate consumer setup. + +Below is an example using Python: + +**Produce messages**: To send messages to an Apache Kafka topic: + +```python +from kafka import KafkaProducer + +# Set up the Kafka producer +producer = KafkaProducer( + bootstrap_servers='your-kafka-host:port', # Replace with your Kafka service host and port + security_protocol='SSL', + ssl_cafile='path/to/ca.pem', + ssl_certfile='path/to/service.cert', + ssl_keyfile='path/to/service.key' +) + +# Send a test message to a topic +producer.send('your-topic', b'This is a test message') +producer.flush() +producer.close() +``` + +**Consume messages**: To read messages from an Apache Kafka topic. + +from kafka import KafkaConsumer + +```python +# Set up the Kafka consumer +consumer = KafkaConsumer( + 'your-topic', # Replace with your topic name + bootstrap_servers='your-kafka-host:port', # Replace with your Kafka service host and port + security_protocol='SSL', + ssl_cafile='path/to/ca.pem', + ssl_certfile='path/to/service.cert', + ssl_keyfile='path/to/service.key', + auto_offset_reset='earliest' +) + +# Consume and print messages from the topic +for message in consumer: + print(f"Received message: {message.value}") +``` ## Integration with Aiven for Apache Kafka Aiven for Apache Kafka® supports a wide range of integrations that extend its capabilities, allowing you to seamlessly connect with external systems. Some key integration options include: -- [Apache Kafka Connect](/docs/products/kafka/kafka-connect): Integrate - with external systems like databases and file systems. +- [Apache Kafka Connect](/docs/products/kafka/kafka-connect): Integrate Kafka with + external systems like databases, file systems, and cloud services, enabling data + import and export. - [Apache Kafka MirrorMaker](/docs/products/kafka/kafka-mirrormaker): - Enable cross-cluster replication. + Enable cross-cluster replication, allowing you to replicate data between Apache Kafka + clusters for disaster recovery or geographical distribution. - [Prometheus and Datadog](/docs/products/kafka/howto/kafka-prometheus-privatelink): - Monitor and manage Kafka metrics. + Monitor and manage Apache Kafka metrics, helping you maintain the health and performance + of your Aiven for Kafka services. - [Elasticsearch](/docs/integrations/send-logs-to-elasticsearch) and OpenSearch: Store - logs and perform search and analytics. + Apache Kafka logs and perform advanced search and analytics, making it easier to + query and analyze your Apache Kafka data. - [Apache Flink® and Apache Kafka® Streams](/docs/products/kafka/howto/kafka-streams-with-aiven-for-kafka): - Process and analyze real-time data streams. + Process and analyze real-time data streams directly from Aiven for Apache Kafka, + enabling complex event processing and stream transformations. These integrations can be configured within the Aiven Console under **Integration endpoints**, allowing you to connect your Kafka service with other platforms and tools efficiently. +## Configure Aiven for Apache Kafka® Connect + +Aiven for Apache Kafka® Connect lets you integrate your Kafka service with various data sources and sinks, enabling you to build data pipelines. + +:::note +Ensure that your Aiven for Kafka service is running on a business or premium plan. +::: + +- For a cost-effective option, [enable Kafka Connect on the same node](/docs/products/kafka/kafka-connect/howto/enable-connect). +- For improved performance and stability, [deploy a dedicated Kafka Connect service](/docs/products/kafka/kafka-connect/get-started#apache_kafka_connect_dedicated_cluster). + +## Enable and use Karapace + +[Karapace](/docs/products/kafka/karapace), an Aiven-built open-source schema registry +for Apache Kafka®, provides schema management and REST APIs for interacting with +Aiven for Apache Kafka. + +1. Enable Karapace schema registry and REST APIs in your Kafka service. +1. Use Karapace to manage schemas and access Kafka via REST APIs. + +Learn more about the [Karapace schema registry](/docs/products/kafka/karapace/concepts/schema-registry-authorization). + ## View topic catalog The Aiven for Apache Kafka® topic catalog provides a centralized interface to view, @@ -206,28 +326,8 @@ they meet governance standards. Discover more in the [Aiven for Apache Kafka® governance overview](/docs/products/kafka/concepts/governance-overview). -## Configure Aiven for Apache Kafka® Connect - -Aiven for Apache Kafka® Connect lets you integrate your Aiven for Apache Kafka -service with other data sources and sinks. Set up Apache Kafka Connect in your project -to start building data pipelines. - -1. Ensure your project includes a Kafka service. -1. Create a dedicated Kafka Connect service via the Aiven Console. -1. Set up and deploy connectors to integrate with external systems. - -## Enable and use Karapace - -Karapace provides schema registry and REST APIs for managing schemas and interacting -with Aiven for Apache Kafka. - -1. Enable Karapace schema registry and REST APIs in your Kafka service. -2. Use Karapace to manage schemas and access Kafka via REST APIs. - -Learn more about [Karapace schema registry](/docs/products/kafka/karapace/concepts/schema-registry-authorization). - -## Next steps +## Related pages - Explore our [examples project](https://github.com/aiven/aiven-examples) for code samples. - Use our [sample data generator project](https://github.com/aiven/python-fake-data-producer-for-apache-kafka) to create test data. From 91b9b6ad776d63424dfdf492547dba580b678970 Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Thu, 29 Aug 2024 15:29:47 +0200 Subject: [PATCH 3/6] fix: terraform kafka version --- docs/products/kafka/get-started.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/products/kafka/get-started.md b/docs/products/kafka/get-started.md index 165798e61..ad11e8d8e 100644 --- a/docs/products/kafka/get-started.md +++ b/docs/products/kafka/get-started.md @@ -65,7 +65,7 @@ the process with Terraform. maintenance_window_time = "23:00:00" kafka_user_config { - kafka_version = "2.7" + kafka_version = "3.7" } } From 7e80b860a55738a00e60c307a066c87b5716ccd4 Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Tue, 10 Sep 2024 13:20:21 +0200 Subject: [PATCH 4/6] update: content --- docs/products/kafka/get-started.md | 197 +++++++++++++++++++---------- 1 file changed, 128 insertions(+), 69 deletions(-) diff --git a/docs/products/kafka/get-started.md b/docs/products/kafka/get-started.md index ad11e8d8e..1afe2c0fb 100644 --- a/docs/products/kafka/get-started.md +++ b/docs/products/kafka/get-started.md @@ -8,19 +8,23 @@ import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import ConsoleLabel from "@site/src/components/ConsoleIcons" import CreateService from "@site/static/includes/create-service-console.md" +import LimitedBadge from "@site/src/components/Badges/LimitedBadge"; +import EarlyBadge from "@site/src/components/Badges/EarlyBadge"; Start using Aiven for Apache Kafka® by setting up and configuring a service, connecting to it, and managing your data. ## Prerequisites -- Access to the [Aiven Console](https://console.aiven.io) +Before you begin, ensure you have access to the following tools: + +- [Aiven Console](https://console.aiven.io) - [Aiven Provider for Terraform](https://registry.terraform.io/providers/aiven/aiven/latest/docs) - [Aiven CLI](/docs/tools/cli) ## Create an Aiven for Apache Kafka® service -You can create your Kafka service using the Aiven Console, the Aiven CLI, or automate -the process with Terraform. +You can create your Aiven for Apache Kafka service using the Aiven Console, the Aiven CLI, +or automate the process with Terraform. @@ -102,7 +106,7 @@ the process with Terraform. terraform apply --auto-approve ``` -1. Store the Terraform outputs in environment variables to use them for +1. Store the Terraform outputs in environment variables to use them when [connecting](#connect-and-produceconsume-messages): ```bash @@ -139,6 +143,9 @@ Parameters: ## Create an Apache Kafka® topic +Once your service is created, you can add topics for organizing and managing your +messages. + @@ -151,6 +158,44 @@ Parameters: These can be changed later. 1. Click **Create topic**. + + + +1. Add the following Terraform configuration to your `.tf` file: + + ```hcl + resource "aiven_kafka_topic" "example_topic" { + project = data.aiven_project.example_project.project + service_name = aiven_kafka.example_kafka.service_name + topic_name = "example-topic" + partitions = 5 + replication = 3 + termination_protection = true + + config { + flush_ms = 10 + cleanup_policy = "compact,delete" + } + + timeouts { + create = "1m" + read = "5m" + } + } + ``` + + Parameters: + - `project`: The name of the project. + - `service_name`: The name of the Aiven for Apache Kafka service. + - `topic_name`: The name of the Kafka topic. + - `partitions`: The number of partitions for the topic. + - `replication`: The replication factor for the topic. + - `termination_protection`: Enables or disables deletion protection for the topic. + - `config`: Additional Kafka settings, such as flush interval and cleanup policy. + - `timeouts`: Optional timeouts for creating and reading the topic. + +1. Run `terraform apply` to create the topic with the defined configurations. + @@ -169,45 +214,48 @@ Parameters: Parameters: - `avn service topic-create`: Creates a topic. - - `--project demo-kafka-project`: Specifies the project name. - - `--service-name demo-kafka-service`: Specifies the Aiven for Apache Kafka® service name. - - `--topic exampleTopic`: Specifies the name of the topic to create. - - `--partitions 2`: Specifies the number of partitions for the topic. - - `--replication 2`: Specifies the replication factor for the topic. + - `--project demo-kafka-project`: The name of the project in Aiven. + - `--service-name demo-kafka-service`: The name of the Aiven for Apache Kafka® service. + - `--topic exampleTopic`: The name of the topic to create. + - `--partitions 2`: The number of partitions for the topic. + - `--replication 2`: The replication factor for the topic. -## Connect and produce/consume messages {#connect-and-produceconsume-messages} - -Connect to your Aiven for Apache Kafka service to start producing and consuming messages. -The **Quick connect** in the Aiven Console helps you establish a connection. +## Connect to your Aiven for Apache Kafka service -### Connect to your Aiven for Apache Kafka service +You can connect to your Aiven for Apache Kafka service to interact with Apache Kafka +topics, allowing you to produce and consume messages. +The **Quick connect** feature in the Aiven Console helps guide you through the +connection process. 1. Log in to the [Aiven Console](https://console.aiven.io/), select your project, and - the Aiven for Apache Kafka® service. -1. On the page of your service, click - **Quick connect**. -1. In the **Connect** window, choose your preferred tool or language, and follow the - connection instructions provided. + the Aiven for Apache Kafka service. +1. On the page, click **Quick connect**. +1. In the **Connect** window, select your preferred tool or language from the drop-down + list. +1. Follow the connection instructions. You may need to download: + + - **[CA certificate](/docs/platform/concepts/tls-ssl-certificates#download-ca-certificates)**: + Required for secure communication. + - **Access certificate** and **Access key**: Needed for client certificate + authentication. 1. Select your authentication method: - **Client certificate**: Use this method if your environment requires secure communication using SSL/TLS certificates. For information on downloading CA certificates, see [TLS/SSL certificates](https://aiven.io/docs/platform/concepts/tls-ssl-certificates#download-ca-certificates). - **SASL**: Use this method if your environment uses Simple Authentication and Security Layer (SASL) for authentication. -1. Follow the connection instructions provided based on the chosen method. 1. Click **Done**. -### Produce and consume messages +## Produce and consume messages -After connecting, use your Apache Kafka client to produce messages to your -Apache Kafka® topics and consume messages using the appropriate consumer setup. +After connecting, use your preferred Apache Kafka client to produce and consume messages. -Below is an example using Python: +### Produce messages -**Produce messages**: To send messages to an Apache Kafka topic: +To produce messages using Python: ```python from kafka import KafkaProducer @@ -227,11 +275,13 @@ producer.flush() producer.close() ``` -**Consume messages**: To read messages from an Apache Kafka topic. +### Consume messages -from kafka import KafkaConsumer +To consume messages using Python: ```python +from kafka import KafkaConsumer + # Set up the Kafka consumer consumer = KafkaConsumer( 'your-topic', # Replace with your topic name @@ -248,56 +298,66 @@ for message in consumer: print(f"Received message: {message.value}") ``` -## Integration with Aiven for Apache Kafka +## Integrate Aiven for Apache Kafka with external systems -Aiven for Apache Kafka® supports a wide range of integrations that extend its -capabilities, allowing you to seamlessly connect with external systems. Some key integration options include: +Aiven for Apache Kafka supports a wide range of integrations, allowing you to connect +with external systems. Key integration options include: -- [Apache Kafka Connect](/docs/products/kafka/kafka-connect): Integrate Kafka with - external systems like databases, file systems, and cloud services, enabling data - import and export. -- [Apache Kafka MirrorMaker](/docs/products/kafka/kafka-mirrormaker): - Enable cross-cluster replication, allowing you to replicate data between Apache Kafka - clusters for disaster recovery or geographical distribution. +- [Apache Kafka Connect](/docs/products/kafka/kafka-connect): Integrates Apache Kafka + with external systems for data import, export, and building complex data pipelines. + [Learn more about configuring Kafka Connect](#configure-aiven-for-apache-kafka-connect). +- [Apache Kafka MirrorMaker](/docs/products/kafka/kafka-mirrormaker): Enables + cross-cluster replication to replicate data between Apache Kafka clusters for disaster + recovery or geographical distribution. - [Prometheus and Datadog](/docs/products/kafka/howto/kafka-prometheus-privatelink): - Monitor and manage Apache Kafka metrics, helping you maintain the health and performance - of your Aiven for Kafka services. -- [Elasticsearch](/docs/integrations/send-logs-to-elasticsearch) and OpenSearch: Store - Apache Kafka logs and perform advanced search and analytics, making it easier to - query and analyze your Apache Kafka data. + Monitors and manages Apache Kafka metrics to help maintain the health and performance + of your Apache Kafka services. +- [Elasticsearch](/docs/integrations/send-logs-to-elasticsearch) and OpenSearch: Stores + Apache Kafka logs and performs advanced search and analytics, making it easier to query + and analyze your Apache Kafka data. - [Apache Flink® and Apache Kafka® Streams](/docs/products/kafka/howto/kafka-streams-with-aiven-for-kafka): - Process and analyze real-time data streams directly from Aiven for Apache Kafka, - enabling complex event processing and stream transformations. + Processes and analyzes real-time data streams from Apache Kafka for complex event + processing and stream transformations. -These integrations can be configured within the Aiven Console -under **Integration endpoints**, allowing you to connect your Kafka service -with other platforms and tools efficiently. +You can configure these integrations in the Aiven Console. In your project, click + to efficient connect your Apache Kafka +service with other platforms and tools. -## Configure Aiven for Apache Kafka® Connect +## Configure Aiven for Apache Kafka® Connect {#configure-aiven-for-apache-kafka-connect} -Aiven for Apache Kafka® Connect lets you integrate your Kafka service with various data sources and sinks, enabling you to build data pipelines. +Aiven for Apache Kafka® Connect integrates your Apache Kafka service with various data +sources and sinks, enabling you to build and manage data pipelines. :::note -Ensure that your Aiven for Kafka service is running on a business or premium plan. +Make sure your Aiven for Apache Kafka service is running on a business or premium plan. ::: -- For a cost-effective option, [enable Kafka Connect on the same node](/docs/products/kafka/kafka-connect/howto/enable-connect). -- For improved performance and stability, [deploy a dedicated Kafka Connect service](/docs/products/kafka/kafka-connect/get-started#apache_kafka_connect_dedicated_cluster). +You can deploy Aiven for Apache Kafka Connect in two ways: + +- For a cost-effective setup, + [enable Apache Kafka Connect on the same node](/docs/products/kafka/kafka-connect/howto/enable-connect) + as your Aiven for Apache Kafka service. This is suitable for smaller workloads where + minimizing costs is a priority. +- For better performance and stability, [deploy a dedicated Aiven for Apache Kafka Connect + service](/docs/products/kafka/kafka-connect/get-started#apache_kafka_connect_dedicated_cluster). + This option is recommended for larger workloads or production environments where + reliability is key. -## Enable and use Karapace +## Enable Karapace [Karapace](/docs/products/kafka/karapace), an Aiven-built open-source schema registry for Apache Kafka®, provides schema management and REST APIs for interacting with Aiven for Apache Kafka. -1. Enable Karapace schema registry and REST APIs in your Kafka service. -1. Use Karapace to manage schemas and access Kafka via REST APIs. +1. [Enable](/docs/products/kafka/karapace/howto/enable-karapace) Karapace schema + registry and REST APIs in your Kafka service. +1. Use Karapace to register, update, and version control your schemas. Learn more about the [Karapace schema registry](/docs/products/kafka/karapace/concepts/schema-registry-authorization). -## View topic catalog +## View topic catalog -The Aiven for Apache Kafka® topic catalog provides a centralized interface to view, +The Aiven for Apache Kafka topic catalog provides a centralized interface to view, request, and manage topics across projects. It simplifies topic management and ensures governance within your Aiven for Apache Kafka infrastructure. @@ -305,29 +365,28 @@ governance within your Aiven for Apache Kafka infrastructure. 1. Click **Tools**. 1. Select Apache Kafka topic catalog. -Explore more in the [Aiven for Apache Kafka® topic catalog](/docs/products/kafka/concepts/topic-catalog-overview). +Learn more about the [Aiven for Apache Kafka® topic catalog](/docs/products/kafka/concepts/topic-catalog-overview). -## Enable governance +## Enable governance -Governance in Aiven for Apache Kafka® provides a secure and efficient way to manage your -Apache Kafka clusters. It centralizes control through policies and roles, ensuring -compliance and enforcing standards across your Aiven for Apache Kafka environments. +[Governance](/docs/products/kafka/concepts/governance-overview) in Aiven for Apache Kafka +provides a secure and efficient way to manage your Apache Kafka clusters. It centralizes +control through policies and roles, ensuring compliance and enforcing standards across +your Aiven for Apache Kafka environments. :::note -You need super admin permissions to enable governance. +You need [super admin](/docs/platform/howto/make-super-admin) permissions to enable +governance. ::: 1. Access the [Aiven Console](https://console.aiven.io/) and click **Admin**. 1. On the organization page, click . 1. Click **Enable governance**. -Manage topic creation and ownership requests for Apache Kafka® resources, ensuring -they meet governance standards. - -Discover more in the [Aiven for Apache Kafka® governance overview](/docs/products/kafka/concepts/governance-overview). - +Learn more about the [Aiven for Apache Kafka® governance overview](/docs/products/kafka/concepts/governance-overview). ## Related pages -- Explore our [examples project](https://github.com/aiven/aiven-examples) for code samples. -- Use our [sample data generator project](https://github.com/aiven/python-fake-data-producer-for-apache-kafka) to create test data. +- Explore [examples project](https://github.com/aiven/aiven-examples) for code samples. +- Use [sample data generator project](https://github.com/aiven/python-fake-data-producer-for-apache-kafka) + to create test data. From 4131af356e3995d108798c2111024e8bfab3eb5c Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Tue, 10 Sep 2024 13:26:15 +0200 Subject: [PATCH 5/6] fix: cross-link --- docs/products/kafka/get-started.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/products/kafka/get-started.md b/docs/products/kafka/get-started.md index 1afe2c0fb..4dd49354f 100644 --- a/docs/products/kafka/get-started.md +++ b/docs/products/kafka/get-started.md @@ -106,8 +106,7 @@ or automate the process with Terraform. terraform apply --auto-approve ``` -1. Store the Terraform outputs in environment variables to use them when - [connecting](#connect-and-produceconsume-messages): +1. Store the Terraform outputs in environment variables to use them when [connecting](#connect): ```bash KAFKA_HOST="$(terraform output -raw kafka_service_host)" @@ -223,7 +222,7 @@ messages. -## Connect to your Aiven for Apache Kafka service +## Connect to your Aiven for Apache Kafka service {#connect} You can connect to your Aiven for Apache Kafka service to interact with Apache Kafka topics, allowing you to produce and consume messages. From a92b1c8f4a7ff2468db9ff522469ef2c0362aee6 Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Tue, 19 Nov 2024 11:42:58 +0100 Subject: [PATCH 6/6] update: address feedback --- docs/products/kafka/get-started.md | 60 +++++++++++++++++------------- 1 file changed, 34 insertions(+), 26 deletions(-) diff --git a/docs/products/kafka/get-started.md b/docs/products/kafka/get-started.md index 4dd49354f..16a5a5f4d 100644 --- a/docs/products/kafka/get-started.md +++ b/docs/products/kafka/get-started.md @@ -63,7 +63,7 @@ or automate the process with Terraform. resource "aiven_kafka" "kafka" { project = var.aiven_project_name cloud_name = "google-europe-west1" - plan = "startup-16" + plan = "startup-2" service_name = "my-kafka" maintenance_window_dow = "friday" maintenance_window_time = "23:00:00" @@ -126,16 +126,19 @@ avn service create SERVICE_NAME \ --cloud CLOUD_REGION \ --plan SERVICE_PLAN \ -c kafka_connect=true \ + -c tiered_storage.enabled=true \ --disk-space-gib STORAGE_SIZE_GIB ``` Parameters: -- `SERVICE_NAME`: The name for your Aiven for Apache Kafka service. -- `CLOUD_REGION`: The cloud region where the service is deployed. For example, +- `SERVICE_NAME`: Name for your Aiven for Apache Kafka service. +- `CLOUD_REGION`: Cloud region for deployment. For example, `google-europe-west3`. -- `SERVICE_PLAN`: The Aiven subscription plan. For example, `business-4`. -- `STORAGE_SIZE_GIB`: The total disk space for data storage in GiB. For example, `600`. +- `SERVICE_PLAN`: Aiven subscription plan. For example, `business-4`. +- `kafka_connect`: Enables Kafka Connect. Use `true` to enable. +- `tiered_storage.enabled`: Enables tiered storage. Use `true` to enable. +- `STORAGE_SIZE_GIB`: Disk space in GiB. For example, `600`. @@ -184,13 +187,15 @@ messages. ``` Parameters: - - `project`: The name of the project. - - `service_name`: The name of the Aiven for Apache Kafka service. - - `topic_name`: The name of the Kafka topic. - - `partitions`: The number of partitions for the topic. - - `replication`: The replication factor for the topic. + - `project`: Name of the project. + - `service_name`: Name of the Aiven for Apache Kafka service. + - `topic_name`: Name of the Apache Kafka topic. + - `partitions`: Number of partitions for the topic. + - `replication`: Replication factor for the topic. - `termination_protection`: Enables or disables deletion protection for the topic. - - `config`: Additional Kafka settings, such as flush interval and cleanup policy. + - `config`: Additional Apache Kafka settings, such as flush interval and + cleanup policy. For a list of supported configurations, see + [Aiven for Apache Kafka topic configurations](https://registry.terraform.io/providers/aiven/aiven/latest/docs/resources/kafka#nested-schema-for-kafka_user_configkafka). - `timeouts`: Optional timeouts for creating and reading the topic. 1. Run `terraform apply` to create the topic with the defined configurations. @@ -226,8 +231,7 @@ messages. You can connect to your Aiven for Apache Kafka service to interact with Apache Kafka topics, allowing you to produce and consume messages. -The **Quick connect** feature in the Aiven Console helps guide you through the -connection process. +Use **Quick connect** in the Aiven Console to guide the process. 1. Log in to the [Aiven Console](https://console.aiven.io/), select your project, and the Aiven for Apache Kafka service. @@ -240,12 +244,12 @@ connection process. Required for secure communication. - **Access certificate** and **Access key**: Needed for client certificate authentication. -1. Select your authentication method: - - **Client certificate**: Use this method if your environment requires secure - communication using SSL/TLS certificates. For information on downloading CA - certificates, see [TLS/SSL certificates](https://aiven.io/docs/platform/concepts/tls-ssl-certificates#download-ca-certificates). - - **SASL**: Use this method if your environment uses Simple Authentication and - Security Layer (SASL) for authentication. +1. Select your authentication method based on your environment: + - **Client certificate**: Provides secure communication using SSL/TLS certificates. + For information on downloading CA certificates, see + [TLS/SSL certificates](https://aiven.io/docs/platform/concepts/tls-ssl-certificates#download-ca-certificates). + - **SASL**: Provides authentication using Simple Authentication and Security Layer + (SASL). 1. Click **Done**. ## Produce and consume messages @@ -335,12 +339,11 @@ You can deploy Aiven for Apache Kafka Connect in two ways: - For a cost-effective setup, [enable Apache Kafka Connect on the same node](/docs/products/kafka/kafka-connect/howto/enable-connect) - as your Aiven for Apache Kafka service. This is suitable for smaller workloads where - minimizing costs is a priority. -- For better performance and stability, [deploy a dedicated Aiven for Apache Kafka Connect - service](/docs/products/kafka/kafka-connect/get-started#apache_kafka_connect_dedicated_cluster). - This option is recommended for larger workloads or production environments where - reliability is key. + as your Aiven for Apache Kafka service. Suitable only for testing or hobbyist workloads + where cost minimization is a priority. +- For better performance and stability, + [deploy a dedicated Aiven for Apache Kafka Connect service](/docs/products/kafka/kafka-connect/get-started#apache_kafka_connect_dedicated_cluster). + Recommended for production deployments and larger workloads requiring reliability. ## Enable Karapace @@ -387,5 +390,10 @@ Learn more about the [Aiven for Apache Kafka® governance overview](/docs/produc ## Related pages - Explore [examples project](https://github.com/aiven/aiven-examples) for code samples. -- Use [sample data generator project](https://github.com/aiven/python-fake-data-producer-for-apache-kafka) +- Use the [sample data generator project](https://github.com/aiven/python-fake-data-producer-for-apache-kafka) to create test data. +- Learn about the [Karapace Schema Registry](https://github.com/Aiven-Open/karapace) for + managing schemas and interacting with Apache Kafka. +- Use the [Apache Kafka REST API](/docs/products/kafka/concepts/kafka-rest-api) to + programmatically access your Apache Kafka service, produce and consume messages, and + manage schemas.