From 303342eb100b70222479457d009f535b0a03dd20 Mon Sep 17 00:00:00 2001 From: Jingsong Date: Tue, 23 Apr 2024 10:54:32 +0800 Subject: [PATCH] [doc] Recommended Engines --- docs/content/engines/overview.md | 60 +++++++++++++++++++++++++------- 1 file changed, 48 insertions(+), 12 deletions(-) diff --git a/docs/content/engines/overview.md b/docs/content/engines/overview.md index 78c4009b6a68..ed80ab622eb2 100644 --- a/docs/content/engines/overview.md +++ b/docs/content/engines/overview.md @@ -28,18 +28,54 @@ under the License. ## Compatibility Matrix -| Engine | Version | Batch Read | Batch Write | Create Table | Alter Table | Streaming Write | Streaming Read | Batch Overwrite | -|:-------------------------------------------------------------------------------:|:-------------:|:----------:|:-----------:|:------------:|:-----------:|:---------------:|:--------------:|:---------------:| -| Flink | 1.15 - 1.19 | ✅ | ✅ | ✅ | ✅(1.17+) | ✅ | ✅ | ✅ | -| Spark | 3.1 - 3.5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅(3.3+) | ✅ | -| Hive | 2.1 - 3.1 | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | -| Trino | 422 - 426 | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | -| Trino | 427 - 439 | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | -| Presto | 0.236 - 0.280 | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | -| [StarRocks](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog/) | 3.1+ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | -| [Doris](https://doris.apache.org/docs/lakehouse/multi-catalog/paimon/) | 2.0+ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | - -Recommended versions are Flink 1.17.2, Spark 3.5.0, Hive 2.3.9 +| Engine | Version | Batch Read | Batch Write | Create Table | Alter Table | Streaming Write | Streaming Read | Batch Overwrite | DELETE & UPDATE | MERGE INTO | +|:-------------------------------------------------------------------------------:|:-------------:|:-----------:|:------------:|:------------:|:------------:|:----------------:|:--------------:|:----------------:|:----------------:|:-----------:| +| Flink | 1.15 - 1.19 | ✅ | ✅ | ✅ | ✅(1.17+) | ✅ | ✅ | ✅ | ✅(1.17+) | ❌ | +| Spark | 3.1 - 3.5 | ✅ | ✅(3.3+) | ✅ | ✅ | ✅(3.3+) | ✅(3.3+) | ✅(3.3+) | ✅(3.2+) | ✅(3.2+) | +| Hive | 2.1 - 3.1 | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | +| Trino | 422 - 426 | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | +| Trino | 427 - 439 | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | +| Presto | 0.236 - 0.280 | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | +| [StarRocks](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog/) | 3.1+ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | +| [Doris](https://doris.apache.org/docs/lakehouse/multi-catalog/paimon/) | 2.0+ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | + +## Streaming Engines + +### Flink Streaming + +Flink is the most comprehensive streaming computing engine that is widely used for data CDC ingestion and the +construction of streaming pipelines. + +Recommended version is Flink 1.17.2. + +### Spark Streaming + +You can also use Spark Streaming to build a streaming pipeline. Spark's schema evolution capability will be better +implemented, but you must accept the mechanism of mini-batch. + +## Batch Engines + +### Spark Batch + +Spark Batch is the most widely used batch computing engine. + +Recommended version is Spark 3.4.3. + +### Flink Batch + +Flink Batch is also available, which can make your pipeline more integrated with streaming and batch unified. + +## OLAP Engines + +### StarRocks + +StarRocks is the most recommended OLAP engine with the most advanced integration. + +Recommended version is StarRocks 3.2.6. + +### Other OLAP + +You can also use Doris and Trino and Presto, or, you can just use Spark, Flink and Hive to query Paimon tables. ## Download