chore: doc updates

README.md

@@ -1,25 +1,26 @@
-# 🥝 Kiwi - Seamless Real-Time Data Streaming
+# 🥝 Kiwi - Extensible Real-Time Data Streaming
 
 [![test](https://github.com/rkrishn7/kiwi/actions/workflows/test.yml/badge.svg)](https://github.com/rkrishn7/kiwi/actions/workflows/test.yml) [![check](https://github.com/rkrishn7/kiwi/actions/workflows/check.yml/badge.svg)](https://github.com/rkrishn7/kiwi/actions/workflows/check.yml) [![CircleCI](https://dl.circleci.com/status-badge/img/gh/rkrishn7/kiwi/tree/main.svg?style=shield)](https://dl.circleci.com/status-badge/redirect/gh/rkrishn7/kiwi/tree/main) ![contributions](https://img.shields.io/badge/contributions-welcome-green)
 
-Kiwi is an extensible WebSocket adapter for real-time data streaming. It implements a simple protocol for clients to subscribe to configured sources, ensuring that they stay reactive and up-to-date with the latest data.
+Kiwi is a WebSocket adapter for real-time data streaming. It implements a simple protocol for clients to subscribe to configured sources, while allowing operators to maintain control over the flow of data via [WebAssembly](https://webassembly.org/) (WASM) plugins. Kiwi is designed to be a lightweight, extensible, and secure solution for delivering real-time data to clients, ensuring that they stay reactive and up-to-date with the latest data.
 
-***NOTE***: Kiwi is currently in active development and is not yet ready for production use.
+***NOTE***: Kiwi is currently in active development and is not yet recommended for production use.
 
-- [🥝 Kiwi - Seamless Real-Time Data Streaming](#-kiwi---seamless-real-time-data-streaming)
+- [🥝 Kiwi - Extensible Real-Time Data Streaming](#-kiwi---extensible-real-time-data-streaming)
   - [Features](#features)
   - [Motivation](#motivation)
   - [Getting Started](#getting-started)
   - [Plugins](#plugins)
   - [Protocol](#protocol)
   - [Configuration](#configuration)
   - [Considerations](#considerations)
+  - [Known Limitations](#known-limitations)
 
 ## Features
 
 - **Subscribe with Ease**: Set up subscriptions to various sources with a simple command. Kiwi efficiently routes event data to connected WebSocket clients based on these subscriptions.
-- **Backpressure Management**: Kiwi draws from flow-control concepts used by Reactive Streams. Specifically, clients can emit a `request(n)` signal to control the rate at which they receive events.
 - **Extensible**: Kiwi supports WebAssembly (WASM) plugins to enrich and control the flow of data. Plugins are called with context about the current connection and event, and can be used to control how/when events are forwarded to downstream clients.
+- **Backpressure Management**: Kiwi draws from flow-control concepts used by Reactive Streams. Specifically, clients can emit a `request(n)` signal to control the rate at which they receive events.
 - **Secure**: Kiwi supports TLS encryption and custom client authentication via WASM plugins.
 - **Configuration Reloads**: Kiwi can reload a subset of its configuration at runtime, allowing for dynamic updates to sources without restarting the server.
 
@@ -54,7 +55,7 @@ Next, in the same directory as the `kiwi.yml` file, run the following command to
 docker run -p 8000:8000 -v $(pwd)/kiwi.yml:/etc/kiwi/config/kiwi.yml ghcr.io/rkrishn7/kiwi:main
 ```
 
-Success! Kiwi is now running and ready to accept WebSocket connections on port 8000. You can start interacting with the server by using a WebSocket client utility of your choice (e.g. [wscat](https://www.npmjs.com/package/wscat)). 
+Success! Kiwi is now running and ready to accept WebSocket connections on port 8000. You can start interacting with the server by using a WebSocket client utility of your choice (e.g. [wscat](https://www.npmjs.com/package/wscat)). Refer to the [protocol documentation](./doc/PROTOCOL.md) for details on how to interact with the Kiwi server.
 
 For more examples, please see the [examples](./examples) directory.
 
@@ -70,7 +71,7 @@ There are two types of plugins that Kiwi supports:
 - **Authentication**: Authentication plugins are invoked when a client connects to the server. They are called with context about the current connection and can be used to authenticate the client, potentially rejecting the connection if the client is not authorized to connect.
   - Authentication plugins allow users of Kiwi to enforce custom authentication logic, such as verifying JWT tokens or checking for specific user roles. Additionally, the plugin may return custom context for the connection which is passed downstream to each invocation of the intercept plugin.
 
-For more information on writing and using plugins, please see the [plugin documentation](./doc/PLUGINS.md).
+For more information on writing and using plugins, please see the [plugin documentation](./doc/PLUGIN.md).
 
 ## Protocol
 
@@ -87,3 +88,9 @@ Kiwi is designed as a real-time event notification service, leveraging WebAssemb
 Kiwi excels at handling event-driven communication with efficient backpressure management, making it suitable for real-time messaging and lightweight data transformation tasks. However, users requiring advanced stream processing capabilities—such as complex event processing (CEP), stateful computations, windowing, and aggregation over unbounded datasets—are encouraged to use specialized stream processing systems.
 
 Kiwi is designed to be a part of a broader architecture where it can work in conjunction with such systems, rather than serve as a standalone solution for high-throughput data processing needs.
+
+## Known Limitations
+
+Currently, Kiwi does not leverage balanced consumer groups for Kafka sources. This means that Kiwi does not support automatic load balancing of partitions across multiple instances of Kiwi, thus each instance of Kiwi subscribes to the entire set of partitions for a given topic. As a result, Kiwi may not be suitable for very high-throughput Kafka sources with large numbers of partitions.
+
+In the future, Kiwi may support balanced consumer groups for Kafka sources, enabling parallel processing of partitions across multiple instances of Kiwi. However, this feature is not planned for at this time.

doc/PLUGIN.md

@@ -0,0 +1,35 @@
+# Kiwi Plugins
+
+Kiwi allows operators to create WASM plugins that can be loaded at start-up to extend the functionality of the server.
+
+
+## Creating a Plugin
+
+A plugin is a WebAssembly module that implements a specific interface. Currently, Kiwi offers a Rust SDK to streamline the process of creating plugins
+
+First, bootstrap a new plugin project using the `cargo` command-line tool:
+
+```sh
+cargo new --lib my-kiwi-plugin
+```
+
+Next, in the project folder, run the following command to add the `kiwi-sdk` crate:
+
+```sh
+cargo add kiwi-sdk
+```
+
+Before writing any code, ensure that crate type is specified as `cdylib` in the `Cargo.toml` file. This is necessary to compile a dynamic library that can be loaded by Kiwi at runtime.
+
+```toml
+[lib]
+crate-type = ["cdylib"]
+```
+
+**Note**: The plugin must be built with the  `--target` flag set to `wasm32-wasi`
+
+```sh
+cargo build --target wasm32-wasi
+```
+
+Nice! You're ready to start writing your plugin. Take a look in the [examples](../examples) directory for Kiwi plugin samples.

examples/README.md

@@ -2,6 +2,13 @@
 
 This directory contains examples of running Kiwi in various configurations. Each example is self-contained and includes a `README.md` file with instructions on how to run it.
 
+### Prerequisites
+
+- [Rust](https://www.rust-lang.org/tools/install) - The Rust toolchain is required to build WASM hooks.
+- [Docker](https://docs.docker.com/get-docker/) - Docker is utilized in each example as a simple way to run Kiwi.
+- [npm](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) - The Node.js package manager is required to install [wscat](https://www.npmjs.com/package/wscat), a WebSocket client used to interact with the Kiwi server.
+
 ## Examples
 
 - [Intercept (Simple)](./intercept-simple): A simple example that demonstrates how to write WASM hooks using the Kiwi SDK and load them into the Kiwi runtime.
+- [Authenticate w/ Outbound HTTP](./authenticate-http): An example that demonstrates how to make outbound HTTP requests in the authenticate hook using the Kiwi SDK.

examples/authenticate-http/README.md

@@ -0,0 +1,62 @@
+# Authenticate w/ Outbound HTTP
+
+This example demonstrates how to make outbound HTTP requests in the authenticate hook using the Kiwi SDK.
+
+- [Authenticate w/ Outbound HTTP](#authenticate-w-outbound-http)
+  - [Running the Example](#running-the-example)
+    - [Building the WASM Hook](#building-the-wasm-hook)
+    - [Running Kiwi](#running-kiwi)
+    - [Establishing a Connection](#establishing-a-connection)
+  - [Recap](#recap)
+
+## Running the Example
+
+> **NOTE**: The commands in this example should be run from this directory (`examples/authenticate-http`).
+
+### Building the WASM Hook
+
+The `wasm32-wasi` target is required to build the WASM hook. This target is not installed by default, so it must be added using the following command:
+
+```sh
+rustup target add wasm32-wasi
+```
+
+Once the target is installed, the WASM hook can be built using the following command:
+
+```sh
+cargo build --target wasm32-wasi
+```
+
+This command will produce the WASM hook at `target/wasm32-wasi/debug/authenticate_http.wasm`.
+
+### Running Kiwi
+
+Now that the WASM hook is built, it can be run with Kiwi. The following command will run Kiwi with the WASM hook and the provided configuration file:
+
+```sh
+docker run -p 8000:8000 -v $(pwd)/kiwi.yml:/etc/kiwi/config/kiwi.yml \
+    -v $(pwd)/target/wasm32-wasi/debug/authenticate_http.wasm:/etc/kiwi/hook/authenticate.wasm \
+    ghcr.io/rkrishn7/kiwi:main
+```
+
+### Establishing a Connection
+
+Now we can interact with the Kiwi server at `ws://localhost:8000`. Let's try it out by subscribing to a counter source and emitting some events. First, let's try to connect to the server using `wscat`, without providing an `x-api-key` query parameter:
+
+```sh
+wscat -c ws://127.0.0.1:8000
+```
+
+The connection should be rejected by the server. Now, let's try to connect to the server, making sure to provide an `x-api-key` query parameter:
+
+```sh
+wscat -c ws://127.0.0.1:8000/some-path?x-api-key=secret
+```
+
+Success! The connection should be accepted by the server. Behind the scenes, your WASM hook is making an outbound HTTP request to a mock authentication server to verify the `x-api-key` query parameter.
+
+## Recap
+
+This example demonstrated how to write an authentication hook using the Kiwi SDK and load it into Kiwi for execution. The hook makes an outbound HTTP request to a mock authentication server to verify the `x-api-key` query parameter.
+
+Real-world use cases for this type of hook include verifying JWT tokens, checking for specific user roles, and more.

examples/authenticate-http/kiwi.yml

@@ -0,0 +1,12 @@
+hooks:
+  authenticate: '/etc/kiwi/hook/authenticate.wasm'
+
+sources:
+  - type: counter
+    id: counter1
+    interval_ms: 1000
+    lazy: true
+    min: 0
+
+server:
+  address: '0.0.0.0:8000'

examples/authenticate-http/src/lib.rs

@@ -1,16 +1,23 @@
+//! An example authenticate hook that parses an API key from the request query string
+//! and makes a request to an API to verify the key
+
 use http::request::Builder;
 use kiwi_sdk::hook::authenticate::{authenticate, Outcome};
 use kiwi_sdk::http::{request as http_request, Request};
 
+/// You must use the `#[intercept]` macro to define an intercept hook.
 #[authenticate]
 fn handle(req: Request<()>) -> Outcome {
     let query = match req.uri().query() {
         Some(query) => query,
+        // Returning `Outcome::Reject` instructs Kiwi to reject the connection
         None => return Outcome::Reject,
     };
 
     let parts: Vec<&str> = query.split('&').collect();
 
+    // Parse the query string to find the API key
+    // If the API key is not found, reject the connection
     let key = {
         let mut token = None;
         for (key, value) in parts.iter().map(|part| {
@@ -30,25 +37,24 @@ fn handle(req: Request<()>) -> Outcome {
         }
     };
 
-    println!("Received API Key: {key}");
-
     let request = Builder::new()
         .method("GET")
         .uri("https://example.com")
         .header("x-api-key", key)
         .body(Vec::new())
         .unwrap();
 
+    // Make a request to the API to verify the API key
     match http_request(request) {
         Ok(res) => {
             if res.status() == 200 {
+                // Returning `Outcome::Authenticate` instructs Kiwi to allow the connection to be established.
                 Outcome::Authenticate
             } else {
                 Outcome::Reject
             }
         }
-        Err(err) => {
-            println!("error: {:?}", err);
+        Err(_) => {
             Outcome::Reject
         }
     }

examples/intercept-simple/README.md

@@ -4,7 +4,6 @@ This example highlights how to write a simple WebAssembly (WASM) hook using the
 
 - [Intercept (Simple)](#intercept-simple)
   - [Running the Example](#running-the-example)
-    - [Prerequisites](#prerequisites)
     - [Building the WASM Hook](#building-the-wasm-hook)
     - [Running Kiwi](#running-kiwi)
     - [Interacting with Kiwi](#interacting-with-kiwi)
@@ -14,12 +13,6 @@ This example highlights how to write a simple WebAssembly (WASM) hook using the
 
 > **NOTE**: The commands in this example should be run from this directory (`examples/intercept-simple`).
 
-### Prerequisites
-
-- [Rust](https://www.rust-lang.org/tools/install) - The Rust toolchain is required to build the WASM hook.
-- [Docker](https://docs.docker.com/get-docker/) - Docker is utilized in this example as a simple way to run Kiwi.
-- [npm](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) - The Node.js package manager is required to install [wscat](https://www.npmjs.com/package/wscat), a WebSocket client used to interact with the Kiwi server.
-
 ### Building the WASM Hook
 
 The `wasm32-wasi` target is required to build the WASM hook. This target is not installed by default, so it must be added using the following command:
@@ -48,13 +41,7 @@ docker run -p 8000:8000 -v $(pwd)/kiwi.yml:/etc/kiwi/config/kiwi.yml \
 
 ### Interacting with Kiwi
 
-The Kiwi server is now running with the WASM hook loaded. To interact with it, let's go ahead and install [wscat](https://github.com/websockets/wscat):
-
-```sh
-npm install -g wscat
-```
-
-Awesome! Now we can interact with the Kiwi server at `ws://localhost:8000`. Let's try it out by subscribing to a counter source and emitting some events. First, let's connect to the server using `wscat`:
+Now we can interact with the Kiwi server at `ws://localhost:8000`. Let's try it out by subscribing to a counter source and emitting some events. First, let's connect to the server using `wscat`:
 
 ```sh
 wscat -c ws://127.0.0.1:8000