Skip to content

Commit

Permalink
update README
Browse files Browse the repository at this point in the history
  • Loading branch information
rkrishn7 committed Mar 15, 2024
1 parent 2c8828e commit 7ab9066
Showing 1 changed file with 18 additions and 18 deletions.
36 changes: 18 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@ Kiwi is a WebSocket adapter for real-time data sources. It implements a simple p

- [🥝 Kiwi - Extensible Real-Time Data Streaming](#-kiwi---extensible-real-time-data-streaming)
- [Features](#features)
- [Supported Sources](#supported-sources)
- [Kafka](#kafka)
- [Counter](#counter)
- [Motivation](#motivation)
- [Getting Started](#getting-started)
- [Plugins](#plugins)
- [Sources](#sources)
- [Kafka](#kafka)
- [Counter](#counter)
- [Protocol](#protocol)
- [Configuration](#configuration)
- [Considerations](#considerations)
Expand All @@ -26,21 +26,6 @@ Kiwi is a WebSocket adapter for real-time data sources. It implements a simple p
- **Secure**: Kiwi supports TLS encryption and custom client authentication via WASM plugins.
- **Configuration Reloads**: Kiwi can reload a subset of its configuration at runtime, allowing for dynamic updates to sources and plugin code without restarting the server.

## Supported Sources

### Kafka

Currently, Kiwi primarily supports Kafka as a data source, with plans to support additional sources in the future. Kafka sources are backed by a high-performance Rust Kafka client, [rust-rdkafka](https://github.com/fede1024/rust-rdkafka), and support automatic partition discovery.

Notably, Kiwi does not leverage balanced consumer groups for Kafka sources. Instead, it subscribes to the entire set of partitions for a given topic and invokes the configured intercept plugin, if any, for each event. This has a few implications:

- Kiwi may not be suitable for very high-throughput Kafka topics, as the freshness of events may be impacted by the combination of the high volume of events across partitions and per-event processing time. There are plans to support a deterministic partitioning plugin for clients in the future to address this limitation for supported use cases.
- Event processing cannot be parallelized across multiple instances of Kiwi. If vertical scaling is not sufficient to handle the combined throughput of all configured sources, Kiwi may not be the best fit.

### Counter

Kiwi also includes a simple counter source for testing and demonstration purposes. The counter source emits a monotonically increasing integer at a configurable interval, and is primarily used to demonstrate the behavior of Kiwi with a simple source.

## Motivation

The digital era has increasingly moved towards real-time data and event-driven architectures. Tools like Apache Kafka have set the standard for robust, high-throughput messaging and streaming, enabling applications to process and react to data as it arrives. Kafka, and the ecosystem built around it, excel at ingesting streams of events and providing the backbone for enterprise-level data processing and analytics. However, there's often a disconnect when trying to extend this real-time data paradigm directly to end-users in a web or mobile environment.
Expand Down Expand Up @@ -90,6 +75,21 @@ There are two types of plugins that Kiwi supports:

For more information on writing and using plugins, please see the [plugin documentation](./doc/PLUGIN.md).

## Sources

### Kafka

Currently, Kiwi primarily supports Kafka as a data source, with plans to support additional sources in the future. Kafka sources are backed by a high-performance Rust Kafka client, [rust-rdkafka](https://github.com/fede1024/rust-rdkafka), and support automatic partition discovery.

Notably, Kiwi does not leverage balanced consumer groups for Kafka sources. Instead, it subscribes to the entire set of partitions for a given topic and invokes the configured intercept plugin, if any, for each event. This has a few implications:

- Kiwi may not be suitable for very high-throughput Kafka topics, as the freshness of events may be impacted by the combination of the high volume of events across partitions and per-event processing time. There are plans to support a deterministic partitioning plugin for clients in the future to address this limitation for supported use cases.
- Event processing cannot be parallelized across multiple instances of Kiwi. If vertical scaling is not sufficient to handle the combined throughput of all configured sources, Kiwi may not be the best fit.

### Counter

Kiwi also includes a simple counter source for testing and demonstration purposes. The counter source emits a monotonically increasing integer at a configurable interval, and is primarily used to demonstrate the behavior of Kiwi with a simple source.

## Protocol

Details on the Kiwi protocol can be found in the [protocol documentation](./doc/PROTOCOL.md).
Expand Down

0 comments on commit 7ab9066

Please sign in to comment.