Skip to content

Commit

Permalink
doc: add rate limit config param (#123)
Browse files Browse the repository at this point in the history
  • Loading branch information
raulb authored Aug 12, 2024
1 parent 8cca50f commit 8a14b42
Show file tree
Hide file tree
Showing 2 changed files with 45 additions and 1 deletion.
44 changes: 44 additions & 0 deletions docs/connectors/configuration-parameters/rate-limiting.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
title: "Rate Limiting"
sidebar_position: 2
---

Destination connectors can be configured to limit the rate at which records can be written. This is especially useful when the destination resource has a rate limit to ensure that the connector does not exceed it. By default, Conduit does not limit the rate at which records are written.

## Configuration parameters

* `sdk.rate.perSecond`: Maximum number of records written per second (0 means no rate limit).
* `sdk.rate.burst`: Allow bursts of at most X records (0 or less means that bursts are not limited). Only takes effect if a rate limit per second is set. Note that if `sdk.batch.size` is bigger than `sdk.rate.burst`, the effective batch size will be equal to `sdk.rate.burst`.

## Example

The pipeline will generate structured records as fast as possible, and write them to the log. When you run it, you'll notice that by using both `sdk.rate.perSecond` and `sdk.rate.burst`, the log destination connector will limit the rate of processed records 8 records per second in batches of 2 (i.e. 2 records every 250ms).

```yaml
version: 2.2
pipelines:
- id: generator-to-log
status: running
description: >
Example pipeline using the generator source connector and the log destination connector.
Showing how to limit the rate of processing records.
connectors:
- id: example
type: source
plugin: generator
settings:
rate: 0 # generating source records as fast as possible
format.type: structured
format.options.id: int
format.options.name: string
operations: create
- id: log
type: destination
plugin: log
settings:
level: info

# limit the rate of proccessing records
sdk.rate.perSecond: 8
sdk.rate.burst: 2
```
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Schema Extraction"
sidebar_position: 2
sidebar_position: 3
---

Source and destination connectors can be configured to automatically extract the
Expand Down

0 comments on commit 8a14b42

Please sign in to comment.