update(clickhouse): kafka table engine support (#264)

aiven · Jun 17, 2024 · 022f745 · 022f745
1 parent 61ebd04
commit 022f745
Show file tree

Hide file tree

Showing 3 changed files with 68 additions and 57 deletions.
diff --git a/docs/products/clickhouse/howto/integrate-kafka.md b/docs/products/clickhouse/howto/integrate-kafka.md
@@ -42,7 +42,7 @@ The following variables will be used later in the code snippets:
 | `KAFKA_SERVICE_NAME`      | Name of the Apache Kafka service you use for the integration.                                              |
 | `PROJECT`                 | Name of Aiven project where your services are located.                                                     |
 | `CONNECTOR_TABLE_NAME`    | Name of the Kafka engine virtual table that is used as a connector.                                        |
-| [DATA_FORMAT`             | Input/output data format in which data is accepted into Aiven for ClickHouse. See [Reference](#reference). |
+| `DATA_FORMAT`             | Input/output data format in which data is accepted into Aiven for ClickHouse. See [Reference](#reference). |
 | `CONSUMER_GROUP_NAME`     | Name of the consumer group. Each message is delivered once per consumer group.                             |
 
 ## Create an integration
@@ -73,11 +73,11 @@ For each table, there are mandatory and optional setting to be defined.
 
 ### Mandatory settings
 
-For each table, you need to define the following:
+For each table, define the following:
 
 -   `name` - name of the connector table
 -   `columns` - array of columns, with names and types
--   `topics` - array of topics, where you want to bring the data from
+-   `topics` - array of topics, where to bring the data from
 -   `data_format` - your preferred format for data input, see
     [Formats for ClickHouse®-Kafka® data exchange](/docs/products/clickhouse/reference/supported-input-output-formats)
 -   `group_name` - consumer group name, that will be created on your
@@ -109,7 +109,7 @@ For each table, you can define the following optional settings:
 | `auto_offset_reset`      | Action to take when there is no initial offset in the offset store or the desired offset is out of range | `earliest` | `smallest`, `earliest`, `beginning`, `largest`, `latest`, `end` | \--     | \--             |
 | `date_time_input_format` | Method to read `DateTime` from text input formats                                                        | `basic`    | `basic`, `best_effort`, `best_effort_us`                        | \--     | \--             |
 | `handle_error_mode`      | Method to handle errors for the Kafka engine                                                             | `default`  | `default`, `stream`                                             | \--     | \--             |
-| `max_block_size`         | Number of rows collected by poll(s) for flushing data from Kafka                                         | `0`        | `0` - `1_000_000_000`                                           | `0`     | `1_000_000_000` |
+| `max_block_size`         | Number of rows collected by a poll for flushing data from Kafka                                         | `0`        | `0` - `1_000_000_000`                                           | `0`     | `1_000_000_000` |
 | `max_rows_per_message`   | Maximum number of rows produced in one Kafka message for row-based formats                               | `1`        | `1` - `1_000_000_000`                                           | `1`     | `1_000_000_000` |
 | `num_consumers`          | Number of consumers per table per replica                                                                | `1`        | `1` - `10`                                                      | `1`     | `10`            |
 | `poll_max_batch_size`    | Maximum amount of messages to be polled in a single Kafka poll                                           | `0`        | `0` - `1_000_000_000`                                           | `0`     | `1_000_000_000` |

diff --git a/docs/products/clickhouse/reference/limitations.md b/docs/products/clickhouse/reference/limitations.md
@@ -55,13 +55,12 @@ on how to work around specific restrictions.
       <td><p>-</p></td>
     </tr>
     <tr>
-      <td><p>Table engines availability</p></td>
+      <td><p>Table engines support</p></td>
       <td>
         <ul>
-          <li><p>Log engine is not supported in Aiven for ClickHouse.</p></li>
           <li>
             <p>
-              Some special table engines and the Log engine are not supported in
+              Some special table engines are not supported in
               Aiven for ClickHouse.
             </p>
           </li>
@@ -76,38 +75,51 @@ on how to work around specific restrictions.
         </ul>
       </td>
       <td>
-        <ul>
-          <li>
-            <p>
-              For storing data, use the
-              <a href="https://clickhouse.com/docs/en/engines/table-engines/special/buffer/">Buffer engine</a>
-              instead of the Log engine.
-            </p>
-          </li>
-          <li>
-            <p>
-              Use the available table engines listed in
-              <a href="/docs/products/clickhouse/reference/supported-table-engines"><span>Supported table engines in Aiven for ClickHouse</span></a>.
-            </p>
-          </li>
-        </ul>
+        <p>
+          Use the available table engines listed in
+          <a href="/docs/products/clickhouse/reference/supported-table-engines"><span>Supported table engines in Aiven for ClickHouse</span></a>.
+        </p>
       </td>
     </tr>
     <tr>
-      <td><p>Cloud availability</p></td>
-      <td><p>Available on AWS, GCP, and Azure only</p></td>
-      <td><p>Use the available cloud providers.</p></td>
+      <td><p>Log table engine support</p></td>
+      <td>
+        <p>Log engine is not supported in Aiven for ClickHouse.</p>
+      </td>
+      <td>
+        <p>
+          For storing data, use the
+          <a href="https://clickhouse.com/docs/en/engines/table-engines/special/buffer/">Buffer engine</a>
+          instead of the Log engine.
+        </p>
+      </td>
+    </tr>
+    <tr>
+      <td><p>Kafka table engine support</p></td>
+      <td>
+        <p>
+          The Kafka table engine is supported via
+          [integration](/docs/products/clickhouse/howto/integrate-kafka) only,
+          not by creating a table in SQL.
+        </p>
+      </td>
+      <td><p>-</p></td>
     </tr>
     <tr>
       <td><p>Kafka Schema Registry</p></td>
       <td>
         <p>
-          Aiven for ClickHouse doesn't support Kafka Schema Registry, which
-          allows to build stream processing pipelines with schemas.
+          Kafka Schema Registry is supported with Aiven for Apache Kafka® and not with an
+          external Kafka endpoint.
         </p>
       </td>
       <td><p>-</p></td>
     </tr>
+    <tr>
+      <td><p>Cloud availability</p></td>
+      <td><p>Available on AWS, GCP, and Azure only</p></td>
+      <td><p>Use the available cloud providers.</p></td>
+    </tr>
     <tr>
       <td><p>Querying all shards at once</p></td>
       <td>

diff --git a/docs/products/clickhouse/reference/supported-table-engines.md b/docs/products/clickhouse/reference/supported-table-engines.md
@@ -1,11 +1,9 @@
 ---
 title: Supported table engines in Aiven for ClickHouse®
+sidebar_label: Table engines
 ---
 
-This article lists the table engines that are currently supported in
-Aiven for ClickHouse®.
-
-ClickHouse® supports several table engines that define the storage
+Aiven for ClickHouse® supports several table engines that define the storage
 parameters, supported queries, and other aspects of the data tables. For
 details on each table engine, see the [ClickHouse
 documentation](https://clickhouse.com/docs/en/engines/table-engines/).
@@ -20,29 +18,30 @@ same engine.
 
 Aiven for ClickHouse supports the following table engines:
 
-| Engine                                      | Engine family   |
-| ------------------------------------------- | --------------- |
-| `AggregatingMergeTree` (remapped)           | MergeTree       |
-| `Buffer`                                    | Special engines |
-| `CollapsingMergeTree` (remapped)            | MergeTree       |
-| `Dictionary`                                | Special engines |
-| `Distributed`                               | Special engines |
-| `GenerateRandom`                            | Special engines |
-| `GraphiteMergeTree` (remapped)              | MergeTree       |
-| `MaterializedView`                          | Special engines |
-| `Memory`                                    | Special engines |
-| `Merge`                                     | Special engines |
-| `MergeTree` (remapped)                      | MergeTree       |
-| `Null`                                      | Special engines |
-| `ReplacingMergeTree` (remapped)             | MergeTree       |
-| `ReplicatedAggregatingMergeTree` (remapped) | MergeTree       |
-| `ReplicatedCollapsingMergeTree`             | MergeTree       |
-| `ReplicatedGraphiteMergeTree`               | MergeTree       |
-| `ReplicatedMergeTree`                       | MergeTree       |
-| `ReplicatedReplacingMergeTree`              | MergeTree       |
-| `ReplicatedSummingMergeTree`                | MergeTree       |
-| `ReplicatedVersionedCollapsingMergeTree`    | MergeTree       |
-| `Set`                                       | Special engines |
-| `SummingMergeTree` (remapped)               | MergeTree       |
-| `VersionedCollapsingMergeTree` (remapped)   | MergeTree       |
-| `View`                                      | Special engines |
+| Engine                                      | Engine family  |
+| ------------------------------------------- | -------------- |
+| `AggregatingMergeTree` (remapped)           | MergeTree      |
+| `Buffer`                                    | Special engine |
+| `CollapsingMergeTree` (remapped)            | MergeTree      |
+| `Dictionary`                                | Special engine |
+| `Distributed`                               | Special engine |
+| `GenerateRandom`                            | Special engine |
+| `GraphiteMergeTree` (remapped)              | MergeTree      |
+| `KafkaEngine` (via [integration](/docs/products/clickhouse/howto/integrate-kafka) only, not via SQL) | Special engine |
+| `MaterializedView`                          | Special engine |
+| `Memory`                                    | Special engine |
+| `Merge`                                     | Special engine |
+| `MergeTree` (remapped)                      | MergeTree      |
+| `Null`                                      | Special engine |
+| `ReplacingMergeTree` (remapped)             | MergeTree      |
+| `ReplicatedAggregatingMergeTree` (remapped) | MergeTree      |
+| `ReplicatedCollapsingMergeTree`             | MergeTree      |
+| `ReplicatedGraphiteMergeTree`               | MergeTree      |
+| `ReplicatedMergeTree`                       | MergeTree      |
+| `ReplicatedReplacingMergeTree`              | MergeTree      |
+| `ReplicatedSummingMergeTree`                | MergeTree      |
+| `ReplicatedVersionedCollapsingMergeTree`    | MergeTree      |
+| `Set`                                       | Special engine |
+| `SummingMergeTree` (remapped)               | MergeTree      |
+| `VersionedCollapsingMergeTree` (remapped)   | MergeTree      |
+| `View`                                      | Special engine |