From 7ab90660905787ac478380d4115d7546066b5725 Mon Sep 17 00:00:00 2001 From: rkrishn7 Date: Fri, 15 Mar 2024 15:07:15 -0700 Subject: [PATCH] update README --- README.md | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index bf06c92..2593c43 100644 --- a/README.md +++ b/README.md @@ -8,12 +8,12 @@ Kiwi is a WebSocket adapter for real-time data sources. It implements a simple p - [🥝 Kiwi - Extensible Real-Time Data Streaming](#-kiwi---extensible-real-time-data-streaming) - [Features](#features) - - [Supported Sources](#supported-sources) - - [Kafka](#kafka) - - [Counter](#counter) - [Motivation](#motivation) - [Getting Started](#getting-started) - [Plugins](#plugins) + - [Sources](#sources) + - [Kafka](#kafka) + - [Counter](#counter) - [Protocol](#protocol) - [Configuration](#configuration) - [Considerations](#considerations) @@ -26,21 +26,6 @@ Kiwi is a WebSocket adapter for real-time data sources. It implements a simple p - **Secure**: Kiwi supports TLS encryption and custom client authentication via WASM plugins. - **Configuration Reloads**: Kiwi can reload a subset of its configuration at runtime, allowing for dynamic updates to sources and plugin code without restarting the server. -## Supported Sources - -### Kafka - -Currently, Kiwi primarily supports Kafka as a data source, with plans to support additional sources in the future. Kafka sources are backed by a high-performance Rust Kafka client, [rust-rdkafka](https://github.com/fede1024/rust-rdkafka), and support automatic partition discovery. - -Notably, Kiwi does not leverage balanced consumer groups for Kafka sources. Instead, it subscribes to the entire set of partitions for a given topic and invokes the configured intercept plugin, if any, for each event. This has a few implications: - -- Kiwi may not be suitable for very high-throughput Kafka topics, as the freshness of events may be impacted by the combination of the high volume of events across partitions and per-event processing time. There are plans to support a deterministic partitioning plugin for clients in the future to address this limitation for supported use cases. -- Event processing cannot be parallelized across multiple instances of Kiwi. If vertical scaling is not sufficient to handle the combined throughput of all configured sources, Kiwi may not be the best fit. - -### Counter - -Kiwi also includes a simple counter source for testing and demonstration purposes. The counter source emits a monotonically increasing integer at a configurable interval, and is primarily used to demonstrate the behavior of Kiwi with a simple source. - ## Motivation The digital era has increasingly moved towards real-time data and event-driven architectures. Tools like Apache Kafka have set the standard for robust, high-throughput messaging and streaming, enabling applications to process and react to data as it arrives. Kafka, and the ecosystem built around it, excel at ingesting streams of events and providing the backbone for enterprise-level data processing and analytics. However, there's often a disconnect when trying to extend this real-time data paradigm directly to end-users in a web or mobile environment. @@ -90,6 +75,21 @@ There are two types of plugins that Kiwi supports: For more information on writing and using plugins, please see the [plugin documentation](./doc/PLUGIN.md). +## Sources + +### Kafka + +Currently, Kiwi primarily supports Kafka as a data source, with plans to support additional sources in the future. Kafka sources are backed by a high-performance Rust Kafka client, [rust-rdkafka](https://github.com/fede1024/rust-rdkafka), and support automatic partition discovery. + +Notably, Kiwi does not leverage balanced consumer groups for Kafka sources. Instead, it subscribes to the entire set of partitions for a given topic and invokes the configured intercept plugin, if any, for each event. This has a few implications: + +- Kiwi may not be suitable for very high-throughput Kafka topics, as the freshness of events may be impacted by the combination of the high volume of events across partitions and per-event processing time. There are plans to support a deterministic partitioning plugin for clients in the future to address this limitation for supported use cases. +- Event processing cannot be parallelized across multiple instances of Kiwi. If vertical scaling is not sufficient to handle the combined throughput of all configured sources, Kiwi may not be the best fit. + +### Counter + +Kiwi also includes a simple counter source for testing and demonstration purposes. The counter source emits a monotonically increasing integer at a configurable interval, and is primarily used to demonstrate the behavior of Kiwi with a simple source. + ## Protocol Details on the Kiwi protocol can be found in the [protocol documentation](./doc/PROTOCOL.md).