Skip to content

Commit

Permalink
[doc] Move system tables to concepts
Browse files Browse the repository at this point in the history
  • Loading branch information
JingsongLi committed Nov 19, 2024
1 parent 19d7f66 commit f09d9c2
Show file tree
Hide file tree
Showing 6 changed files with 81 additions and 47 deletions.
2 changes: 1 addition & 1 deletion docs/content/concepts/spec/_index.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Specification
bookCollapseSection: true
weight: 6
weight: 7
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
title: "System Tables"
weight: 2
weight: 6
type: docs
aliases:
- /maintenance/system-tables.html
- /concepts/system-tables.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
Expand All @@ -26,9 +26,15 @@ under the License.

# System Tables

## Table Specified System Table
Paimon provides a very rich set of system tables to help users better analyze and query the status of Paimon tables:

Table specified system tables contain metadata and information about each table, such as the snapshots created and the options in use. Users can access system tables with batch queries.
1. Query the status of the data table: Data System Table.
2. Query the global status of the entire Catalog: Global System Table.

## Data System Table

Data System tables contain metadata and information about each Paimon data table, such as the snapshots created and the
options in use. Users can access system tables with batch queries.

Currently, Flink, Spark, Trino and StarRocks support querying system tables.

Expand Down Expand Up @@ -130,6 +136,26 @@ SELECT * FROM my_table$audit_log;
*/
```

### Binlog Table

You can query the binlog through binlog table. In the binlog system table, the update before and update after will be packed in one row.

```sql
SELECT * FROM T$binlog;

/*
+------------------+----------------------+-----------------------+
| rowkind | column_0 | column_1 |
+------------------+----------------------+-----------------------+
| +I | [col_0] | [col_1] |
+------------------+----------------------+-----------------------+
| +U | [col_0_ub, col_0_ua] | [col_1_ub, col_1_ua] |
+------------------+----------------------+-----------------------+
| -D | [col_0] | [col_1] |
+------------------+----------------------+-----------------------+
*/
```

### Read-optimized Table

If you require extreme reading performance and can accept reading slightly old data,
Expand Down Expand Up @@ -347,6 +373,22 @@ SELECT * FROM my_table$buckets;
*/
```

### Statistic Table
You can query the statistic information through statistic table.

```sql
SELECT * FROM T$statistics;

/*
+--------------+------------+-----------------------+------------------+----------+
| snapshot_id | schema_id | mergedRecordCount | mergedRecordSize | colstat |
+--------------+------------+-----------------------+------------------+----------+
| 2 | 0 | 2 | 2 | {} |
+--------------+------------+-----------------------+------------------+----------+
1 rows in set
*/
```

## Global System Table

Global system tables contain the statistical information of all the tables exists in paimon. For convenient of searching, we create a reference system database called `sys`.
Expand Down Expand Up @@ -390,40 +432,3 @@ SELECT * FROM sys.catalog_options;
1 rows in set
*/
```

### Statistic Table
You can query the statistic information through statistic table.

```sql
SELECT * FROM T$statistics;

/*
+--------------+------------+-----------------------+------------------+----------+
| snapshot_id | schema_id | mergedRecordCount | mergedRecordSize | colstat |
+--------------+------------+-----------------------+------------------+----------+
| 2 | 0 | 2 | 2 | {} |
+--------------+------------+-----------------------+------------------+----------+
1 rows in set
*/
```
### Binlog Table

You can query the binlog through binlog table. In the binlog system table, the update before and update after will be packed in one row.

```sql
SELECT * FROM T$binlog;

/*
+------------------+----------------------+-----------------------+
| rowkind | column_0 | column_1 |
+------------------+----------------------+-----------------------+
| +I | [col_0] | [col_1] |
+------------------+----------------------+-----------------------+
| +U | [col_0_ub, col_0_ua] | [col_1_ub, col_1_ua] |
+------------------+----------------------+-----------------------+
| -D | [col_0] | [col_1] |
+------------------+----------------------+-----------------------+
*/
```


33 changes: 31 additions & 2 deletions docs/content/concepts/table-types.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,14 +94,43 @@ CREATE TABLE my_table (

## View

View is supported when the metastore can support view, for example, hive metastore.
View is supported when the metastore can support view, for example, hive metastore. If you don't have metastore, you
can only use temporary View, which only exists in the current session. This chapter mainly describes persistent views.

View will currently save the original SQL. If you need to use View across engines, you can write a cross engine
SQL statement. For example:

{{< tabs "view" >}}
{{< tab "Flink SQL" >}}

```sql
CREATE VIEW my_view AS SELECT a + 1, b FROM my_db.my_source;
CREATE VIEW [IF NOT EXISTS] [catalog_name.][db_name.]view_name
[( columnName [, columnName ]* )] [COMMENT view_comment]
AS query_expression;

DROP VIEW [IF EXISTS] [catalog_name.][db_name.]view_name;

SHOW VIEWS;

SHOW CREATE VIEW my_view;
```
{{< /tab >}}

{{< tab "Spark SQL" >}}

```sql
CREATE [OR REPLACE] VIEW [IF NOT EXISTS] [catalog_name.][db_name.]view_name
[( columnName [, columnName ]* )] [COMMENT view_comment]
AS query_expression;

DROP VIEW [IF EXISTS] [catalog_name.][db_name.]view_name;

SHOW VIEWS;
```

{{< /tab >}}

{{< /tabs >}}

## Format Table

Expand Down
2 changes: 1 addition & 1 deletion docs/content/flink/savepoint.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ to learn how to configure and trigger savepoint.
**Step 3: Choose the tag corresponding to the savepoint.**

The tag corresponding to the savepoint will be named in the form of `savepoint-${savepointID}`. You can refer to
[Tags Table]({{< ref "maintenance/system-tables#tags-table" >}}) to query.
[Tags Table]({{< ref "concepts/system-tables#tags-table" >}}) to query.

**Step 4: Rollback the paimon table.**

Expand Down
2 changes: 1 addition & 1 deletion docs/content/primary-key-table/query-performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ The table schema has the greatest impact on query performance. See [Table Mode](
For Merge On Read table, the most important thing you should pay attention to is the number of buckets, which will limit
the concurrency of reading data.

For MOW (Deletion Vectors) or COW table or [Read Optimized]({{< ref "maintenance/system-tables#read-optimized-table" >}}) table,
For MOW (Deletion Vectors) or COW table or [Read Optimized]({{< ref "concepts/system-tables#read-optimized-table" >}}) table,
There is no limit to the concurrency of reading data, and they can also utilize some filtering conditions for non-primary-key columns.

## Data Skipping By Primary Key Filter
Expand Down
2 changes: 1 addition & 1 deletion docs/content/primary-key-table/table-mode.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ If you don't want to use Deletion Vectors mode, you want to query fast enough in
older data, you can also:

1. Configure 'compaction.optimization-interval' when writing data.
2. Query from [read-optimized system table]({{< ref "maintenance/system-tables#read-optimized-table" >}}). Reading from
2. Query from [read-optimized system table]({{< ref "concepts/system-tables#read-optimized-table" >}}). Reading from
results of optimized files avoids merging records with the same key, thus improving reading performance.

You can flexibly balance query performance and data latency when reading.

0 comments on commit f09d9c2

Please sign in to comment.