Skip to content

Commit

Permalink
[core] Rename paimon: tablestore to paimon
Browse files Browse the repository at this point in the history
  • Loading branch information
JingsongLi committed Mar 18, 2023
1 parent 3eb0df5 commit 19ec6f2
Show file tree
Hide file tree
Showing 108 changed files with 369 additions and 413 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Flink Table Store is developed under the umbrella of [Apache Flink](https://flin
Please check out the full [documentation](https://nightlies.apache.org/flink/paimon-docs-master/), hosted by the
[ASF](https://www.apache.org/), for detailed information and user guides.

Check our [quick-start](https://nightlies.apache.org/flink/paimon-docs-master/docs/try-paimon/quick-start/) guide for simple setup instructions to get you started with the table store.
Check our [quick-start](https://nightlies.apache.org/flink/paimon-docs-master/docs/try-paimon/quick-start/) guide for simple setup instructions to get you started with the paimon.

## Building

Expand Down
4 changes: 2 additions & 2 deletions docs/content/engines/flink.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ tar -xzf flink-*.tgz

**Step 2: Copy Table Store Bundled Jar**

Copy table store bundled jar to the `lib` directory of your Flink home.
Copy paimon bundled jar to the `lib` directory of your Flink home.

```bash
cp paimon-flink-*.jar <FLINK_HOME>/lib/
Expand Down Expand Up @@ -138,7 +138,7 @@ CREATE TEMPORARY TABLE word_table (
'fields.word.length' = '1'
);

-- table store requires checkpoint interval in streaming mode
-- paimon requires checkpoint interval in streaming mode
SET 'execution.checkpointing.interval' = '10 s';

-- write streaming data to dynamic table
Expand Down
12 changes: 6 additions & 6 deletions docs/content/engines/hive.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,13 +70,13 @@ You can find Hive connector jar in `./paimon-hive/paimon-hive-connector-<hive-ve
There are several ways to add this jar to Hive.

* You can create an `auxlib` folder under the root directory of Hive, and copy `paimon-hive-connector-{{< version >}}.jar` into `auxlib`.
* You can also copy this jar to a path accessible by Hive, then use `add jar /path/to/paimon-hive-connector-{{< version >}}.jar` to enable table store support in Hive. Note that this method is not recommended. If you're using the MR execution engine and running a join statement, you may be faced with the exception `org.apache.hive.com.esotericsoftware.kryo.kryoexception: unable to find class`.
* You can also copy this jar to a path accessible by Hive, then use `add jar /path/to/paimon-hive-connector-{{< version >}}.jar` to enable paimon support in Hive. Note that this method is not recommended. If you're using the MR execution engine and running a join statement, you may be faced with the exception `org.apache.hive.com.esotericsoftware.kryo.kryoexception: unable to find class`.

NOTE: If you are using HDFS, make sure that the environment variable `HADOOP_HOME` or `HADOOP_CONF_DIR` is set.

## Quick Start with Table Store Hive Catalog

By using table store Hive catalog, you can create, drop and insert into table store tables from Flink. These operations directly affect the corresponding Hive metastore. Tables created in this way can also be accessed directly from Hive.
By using paimon Hive catalog, you can create, drop and insert into paimon tables from Flink. These operations directly affect the corresponding Hive metastore. Tables created in this way can also be accessed directly from Hive.

**Step 1: Prepare Table Store Hive Catalog Jar File for Flink**

Expand All @@ -88,7 +88,7 @@ Execute the following Flink SQL script in Flink SQL client to define a Table Sto

```sql
-- Flink SQL CLI
-- Define table store Hive catalog
-- Define paimon Hive catalog

CREATE CATALOG my_hive WITH (
'type' = 'paimon',
Expand All @@ -97,11 +97,11 @@ CREATE CATALOG my_hive WITH (
'warehouse' = '/path/to/table/store/warehouse'
);

-- Use table store Hive catalog
-- Use paimon Hive catalog

USE CATALOG my_hive;

-- Create a table in table store Hive catalog (use "default" database by default)
-- Create a table in paimon Hive catalog (use "default" database by default)

CREATE TABLE test_table (
a int,
Expand Down Expand Up @@ -155,7 +155,7 @@ OK

## Quick Start with External Table

To access existing table store table, you can also register them as external tables in Hive. Run the following Hive SQL in Hive CLI.
To access existing paimon table, you can also register them as external tables in Hive. Run the following Hive SQL in Hive CLI.

```sql
-- Assume that paimon-hive-connector-{{< version >}}.jar is already in auxlib directory.
Expand Down
6 changes: 3 additions & 3 deletions docs/content/engines/spark2.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ After the guide, all table files should be stored under the path `/tmp/table_sto

**Step 2: Specify Table Store Jar File**

You can append path to table store jar file to the `--jars` argument when starting `spark-shell`.
You can append path to paimon jar file to the `--jars` argument when starting `spark-shell`.

```bash
spark-shell ... --jars /path/to/paimon-spark-2-{{< version >}}.jar
Expand All @@ -84,10 +84,10 @@ Alternatively, you can copy `paimon-spark-2-{{< version >}}.jar` under `spark/ja

**Step 3: Query Table**

Table store with Spark 2.4 does not support DDL. You can use the `Dataset` reader and register the `Dataset` as a temporary table. In spark shell:
Paimon with Spark 2.4 does not support DDL. You can use the `Dataset` reader and register the `Dataset` as a temporary table. In spark shell:

```scala
val dataset = spark.read.format("tablestore").load("file:/tmp/table_store/default.db/word_count")
val dataset = spark.read.format("paimon").load("file:/tmp/table_store/default.db/word_count")
dataset.createOrReplaceTempView("word_count")
spark.sql("SELECT * FROM word_count").show()
```
16 changes: 8 additions & 8 deletions docs/content/engines/spark3.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ If you are using HDFS, make sure that the environment variable `HADOOP_HOME` or

**Step 1: Specify Table Store Jar File**

Append path to table store jar file to the `--jars` argument when starting `spark-sql`.
Append path to paimon jar file to the `--jars` argument when starting `spark-sql`.

```bash
spark-sql ... --jars /path/to/paimon-spark-3.3-{{< version >}}.jar
Expand All @@ -82,19 +82,19 @@ Alternatively, you can copy `paimon-spark-3.3-{{< version >}}.jar` under `spark/

**Step 2: Specify Table Store Catalog**

When starting `spark-sql`, use the following command to register Table Store’s Spark catalog with the name `tablestore`. Table files of the warehouse is stored under `/tmp/table_store`.
When starting `spark-sql`, use the following command to register Table Store’s Spark catalog with the name `paimon`. Table files of the warehouse is stored under `/tmp/table_store`.

```bash
spark-sql ... \
--conf spark.sql.catalog.tablestore=org.apache.paimon.spark.SparkCatalog \
--conf spark.sql.catalog.tablestore.warehouse=file:/tmp/table_store
--conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
--conf spark.sql.catalog.paimon.warehouse=file:/tmp/table_store
```

After `spark-sql` command line has started, run the following SQL to create and switch to database `tablestore.default`.
After `spark-sql` command line has started, run the following SQL to create and switch to database `paimon.default`.

```sql
CREATE DATABASE tablestore.default;
USE tablestore.default;
CREATE DATABASE paimon.default;
USE paimon.default;
```

**Step 3: Create a table and Write Some Records**
Expand Down Expand Up @@ -142,7 +142,7 @@ spark-shell ... --jars /path/to/paimon-spark-3.3-{{< version >}}.jar
```

```scala
val dataset = spark.read.format("tablestore").load("file:/tmp/table_store/default.db/my_table")
val dataset = spark.read.format("paimon").load("file:/tmp/table_store/default.db/my_table")
dataset.createOrReplaceTempView("my_table")
spark.sql("SELECT * FROM my_table").show()
```
Expand Down
22 changes: 11 additions & 11 deletions docs/content/filesystems/oss.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ under the License.
## Download

[Download](https://repo.maven.apache.org/maven2/org/apache/flink/paimon-oss/{{< version >}}/paimon-oss-{{< version >}}.jar)
flink table store shaded jar.
flink paimon shaded jar.

{{< /stable >}}

Expand Down Expand Up @@ -78,11 +78,11 @@ Place `paimon-oss-{{< version >}}.jar` together with `paimon-spark-{{< version >

```shell
spark-sql \
--conf spark.sql.catalog.tablestore=org.apache.paimon.spark.SparkCatalog \
--conf spark.sql.catalog.tablestore.warehouse=oss://<bucket-name>/ \
--conf spark.sql.catalog.tablestore.fs.oss.endpoint=oss-cn-hangzhou.aliyuncs.com \
--conf spark.sql.catalog.tablestore.fs.oss.accessKeyId=xxx \
--conf spark.sql.catalog.tablestore.fs.oss.accessKeySecret=yyy
--conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
--conf spark.sql.catalog.paimon.warehouse=oss://<bucket-name>/ \
--conf spark.sql.catalog.paimon.fs.oss.endpoint=oss-cn-hangzhou.aliyuncs.com \
--conf spark.sql.catalog.paimon.fs.oss.accessKeyId=xxx \
--conf spark.sql.catalog.paimon.fs.oss.accessKeySecret=yyy
```

{{< /tab >}}
Expand All @@ -94,9 +94,9 @@ NOTE: You need to ensure that Hive metastore can access `oss`.
Place `paimon-oss-{{< version >}}.jar` together with `paimon-hive-connector-{{< version >}}.jar` under Hive's auxlib directory, and start like

```sql
SET tablestore.fs.oss.endpoint=oss-cn-hangzhou.aliyuncs.com;
SET tablestore.fs.oss.accessKeyId=xxx;
SET tablestore.fs.oss.accessKeySecret=yyy;
SET paimon.fs.oss.endpoint=oss-cn-hangzhou.aliyuncs.com;
SET paimon.fs.oss.accessKeyId=xxx;
SET paimon.fs.oss.accessKeySecret=yyy;
```

And read table from hive metastore, table can be created by Flink or Spark, see [Catalog with Hive Metastore]({{< ref "how-to/creating-catalogs" >}})
Expand All @@ -109,9 +109,9 @@ SELECT COUNT(1) FROM test_table;

{{< tab "Trino" >}}

Place `paimon-oss-{{< version >}}.jar` together with `paimon-trino-{{< version >}}.jar` under `plugin/tablestore` directory.
Place `paimon-oss-{{< version >}}.jar` together with `paimon-trino-{{< version >}}.jar` under `plugin/paimon` directory.

Add options in `etc/catalog/tablestore.properties`.
Add options in `etc/catalog/paimon.properties`.
```shell
fs.oss.endpoint=oss-cn-hangzhou.aliyuncs.com
fs.oss.accessKeyId=xxx
Expand Down
22 changes: 11 additions & 11 deletions docs/content/filesystems/s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ under the License.
## Download

[Download](https://repo.maven.apache.org/maven2/org/apache/flink/paimon-s3/{{< version >}}/paimon-s3-{{< version >}}.jar)
flink table store shaded jar.
flink paimon shaded jar.

{{< /stable >}}

Expand Down Expand Up @@ -78,11 +78,11 @@ Place `paimon-s3-{{< version >}}.jar` together with `paimon-spark-{{< version >}

```shell
spark-sql \
--conf spark.sql.catalog.tablestore=org.apache.paimon.spark.SparkCatalog \
--conf spark.sql.catalog.tablestore.warehouse=s3://<bucket>/<endpoint> \
--conf spark.sql.catalog.tablestore.s3.endpoint=your-endpoint-hostname \
--conf spark.sql.catalog.tablestore.s3.access-key=xxx \
--conf spark.sql.catalog.tablestore.s3.secret-key=yyy
--conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
--conf spark.sql.catalog.paimon.warehouse=s3://<bucket>/<endpoint> \
--conf spark.sql.catalog.paimon.s3.endpoint=your-endpoint-hostname \
--conf spark.sql.catalog.paimon.s3.access-key=xxx \
--conf spark.sql.catalog.paimon.s3.secret-key=yyy
```

{{< /tab >}}
Expand All @@ -94,9 +94,9 @@ NOTE: You need to ensure that Hive metastore can access `s3`.
Place `paimon-s3-{{< version >}}.jar` together with `paimon-hive-connector-{{< version >}}.jar` under Hive's auxlib directory, and start like

```sql
SET tablestore.s3.endpoint=your-endpoint-hostname;
SET tablestore.s3.access-key=xxx;
SET tablestore.s3.secret-key=yyy;
SET paimon.s3.endpoint=your-endpoint-hostname;
SET paimon.s3.access-key=xxx;
SET paimon.s3.secret-key=yyy;
```

And read table from hive metastore, table can be created by Flink or Spark, see [Catalog with Hive Metastore]({{< ref "how-to/creating-catalogs" >}})
Expand All @@ -109,9 +109,9 @@ SELECT COUNT(1) FROM test_table;

{{< tab "Trino" >}}

Place `paimon-s3-{{< version >}}.jar` together with `paimon-trino-{{< version >}}.jar` under `plugin/tablestore` directory.
Place `paimon-s3-{{< version >}}.jar` together with `paimon-trino-{{< version >}}.jar` under `plugin/paimon` directory.

Add options in `etc/catalog/tablestore.properties`.
Add options in `etc/catalog/paimon.properties`.
```shell
s3.endpoint=your-endpoint-hostname
s3.access-key=xxx
Expand Down
24 changes: 12 additions & 12 deletions docs/content/how-to/creating-catalogs.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,18 +54,18 @@ USE CATALOG my_catalog;

{{< tab "Spark3" >}}

The following shell command registers a Table Store catalog named `tablestore`. Metadata and table files are stored under `hdfs://path/to/warehouse`.
The following shell command registers a paimon catalog named `paimon`. Metadata and table files are stored under `hdfs://path/to/warehouse`.

```bash
spark-sql ... \
--conf spark.sql.catalog.tablestore=org.apache.paimon.spark.SparkCatalog \
--conf spark.sql.catalog.tablestore.warehouse=hdfs://path/to/warehouse
--conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
--conf spark.sql.catalog.paimon.warehouse=hdfs://path/to/warehouse
```

After `spark-sql` is started, you can switch to the `default` database of the `tablestore` catalog with the following SQL.
After `spark-sql` is started, you can switch to the `default` database of the `paimon` catalog with the following SQL.

```sql
USE tablestore.default;
USE paimon.default;
```

{{< /tab >}}
Expand Down Expand Up @@ -133,20 +133,20 @@ USE CATALOG my_hive;

To enable Table Store Hive catalog support in Spark3, append the path of Table Store Hive catalog jar file to `--jars` argument when starting spark.

The following shell command registers a Table tore Hive catalog named `tablestore`. Metadata and table files are stored under `hdfs://path/to/warehouse`. In addition, metadata is also stored in Hive metastore.
The following shell command registers a Table tore Hive catalog named `paimon`. Metadata and table files are stored under `hdfs://path/to/warehouse`. In addition, metadata is also stored in Hive metastore.

```bash
spark-sql ... \
--conf spark.sql.catalog.tablestore=org.apache.paimon.spark.SparkCatalog \
--conf spark.sql.catalog.tablestore.warehouse=hdfs://path/to/warehouse \
--conf spark.sql.catalog.tablestore.metastore=hive \
--conf spark.sql.catalog.tablestore.uri=thrift://<hive-metastore-host-name>:<port>
--conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
--conf spark.sql.catalog.paimon.warehouse=hdfs://path/to/warehouse \
--conf spark.sql.catalog.paimon.metastore=hive \
--conf spark.sql.catalog.paimon.uri=thrift://<hive-metastore-host-name>:<port>
```

After `spark-sql` is started, you can switch to the `default` database of the `tablestore` catalog with the following SQL.
After `spark-sql` is started, you can switch to the `default` database of the `paimon` catalog with the following SQL.

```sql
USE tablestore.default;
USE paimon.default;
```

{{< /tab >}}
Expand Down
4 changes: 2 additions & 2 deletions docs/content/how-to/creating-tables.md
Original file line number Diff line number Diff line change
Expand Up @@ -342,7 +342,7 @@ CREATE TABLE MyTable (
Spark3 only supports creating external tables through Scala API. The following Scala code loads the table located at `hdfs://path/to/table` into a `DataSet`.

```scala
val dataset = spark.read.format("tablestore").load("hdfs://path/to/table")
val dataset = spark.read.format("paimon").load("hdfs://path/to/table")
```

{{< /tab >}}
Expand All @@ -352,7 +352,7 @@ val dataset = spark.read.format("tablestore").load("hdfs://path/to/table")
Spark2 only supports creating external tables through Scala API. The following Scala code loads the table located at `hdfs://path/to/table` into a `DataSet`.

```scala
val dataset = spark.read.format("tablestore").load("hdfs://path/to/table")
val dataset = spark.read.format("paimon").load("hdfs://path/to/table")
```

{{< /tab >}}
Expand Down
2 changes: 1 addition & 1 deletion docs/content/how-to/lookup-joins.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Table Store supports lookup joins on unpartitioned tables with primary keys in F
First, let's create a Table Store table and update it in real-time.

```sql
-- Create a table store catalog
-- Create a paimon catalog
CREATE CATALOG my_catalog WITH (
'type'='paimon',
'warehouse'='hdfs://nn:8020/warehouse/path' -- or 'file://tmp/foo/bar'
Expand Down
6 changes: 3 additions & 3 deletions docs/content/maintenance/configurations.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,18 +28,18 @@ under the License.

### CoreOptions

Core options for table store.
Core options for paimon.

{{< generated/core_configuration >}}

### CatalogOptions

Options for table store catalog.
Options for paimon catalog.

{{< generated/catalog_configuration >}}

### FlinkConnectorOptions

Flink connector options for table store.
Flink connector options for paimon.

{{< generated/flink_connector_configuration >}}
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
<td><h5>metastore</h5></td>
<td style="word-wrap: break-word;">"filesystem"</td>
<td>String</td>
<td>Metastore of table store catalog, supports filesystem and hive.</td>
<td>Metastore of paimon catalog, supports filesystem and hive.</td>
</tr>
<tr>
<td><h5>table.type</h5></td>
Expand Down
4 changes: 2 additions & 2 deletions docs/layouts/shortcodes/generated/core_configuration.html
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
<td><h5>bucket-key</h5></td>
<td style="word-wrap: break-word;">(none)</td>
<td>String</td>
<td>Specify the table store distribution policy. Data is assigned to each bucket according to the hash value of bucket-key.<br />If you specify multiple fields, delimiter is ','.<br />If not specified, the primary key will be used; if there is no primary key, the full row will be used.</td>
<td>Specify the paimon distribution policy. Data is assigned to each bucket according to the hash value of bucket-key.<br />If you specify multiple fields, delimiter is ','.<br />If not specified, the primary key will be used; if there is no primary key, the full row will be used.</td>
</tr>
<tr>
<td><h5>changelog-producer</h5></td>
Expand Down Expand Up @@ -348,7 +348,7 @@
<td><h5>write-mode</h5></td>
<td style="word-wrap: break-word;">change-log</td>
<td><p>Enum</p></td>
<td>Specify the write mode for table.<br /><br />Possible values:<ul><li>"append-only": The table can only accept append-only insert operations. Neither data deduplication nor any primary key constraints will be done when inserting rows into table store.</li><li>"change-log": The table can accept insert/delete/update operations.</li></ul></td>
<td>Specify the write mode for table.<br /><br />Possible values:<ul><li>"append-only": The table can only accept append-only insert operations. Neither data deduplication nor any primary key constraints will be done when inserting rows into paimon.</li><li>"change-log": The table can accept insert/delete/update operations.</li></ul></td>
</tr>
<tr>
<td><h5>write-only</h5></td>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@
<td><h5>sink.partition-shuffle</h5></td>
<td style="word-wrap: break-word;">false</td>
<td>Boolean</td>
<td>The option to enable shuffle data by dynamic partition fields in sink phase for table store.</td>
<td>The option to enable shuffle data by dynamic partition fields in sink phase for paimon.</td>
</tr>
<tr>
<td><h5>streaming-read-atomic</h5></td>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ case $STARTSTOP in
(start)
log=$BENCHMARK_LOG_DIR/metric-client.log
log_setting=(-Dlog.file="$log" -Dlog4j.configuration=file:"$BENCHMARK_CONF_DIR"/log4j.properties -Dlog4j.configurationFile=file:"$BENCHMARK_CONF_DIR"/log4j.properties)
java "${log_setting[@]}" -cp "$BENCHMARK_HOME/lib/*:$FLINK_HOME/lib/*" org.apache.flink.table.store.benchmark.metric.cpu.CpuMetricSender &
java "${log_setting[@]}" -cp "$BENCHMARK_HOME/lib/*:$FLINK_HOME/lib/*" org.apache.paimon.benchmark.metric.cpu.CpuMetricSender &
;;

(stop)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,4 @@ bin=`cd "$bin"; pwd`
log=$BENCHMARK_LOG_DIR/benchmark.log
log_setting=(-Dlog.file="$log" -Dlog4j.configuration=file:"$BENCHMARK_CONF_DIR"/log4j.properties -Dlog4j.configurationFile=file:"$BENCHMARK_CONF_DIR"/log4j.properties)

java "${log_setting[@]}" -cp "$BENCHMARK_HOME/lib/*:$FLINK_HOME/lib/*" org.apache.flink.table.store.benchmark.Benchmark --location $BENCHMARK_HOME --queries $1 --sinks $2
java "${log_setting[@]}" -cp "$BENCHMARK_HOME/lib/*:$FLINK_HOME/lib/*" org.apache.paimon.benchmark.Benchmark --location $BENCHMARK_HOME --queries $1 --sinks $2
Loading

0 comments on commit 19ec6f2

Please sign in to comment.