Skip to content

Commit

Permalink
[core] support z-order range sort action (apache#1846)
Browse files Browse the repository at this point in the history
  • Loading branch information
leaves12138 authored Aug 23, 2023
1 parent 27b155f commit 0dbcff6
Show file tree
Hide file tree
Showing 20 changed files with 2,572 additions and 53 deletions.
2 changes: 2 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -234,6 +234,8 @@ from http://flink.apache.org/ version 1.17.0

paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkGenericCatalog.java
paimon-hive/paimon-hive-common/src/test/java/org/apache/paimon/hive/TestHiveMetastore.java
paimon-core/src/main/java/org/apache/paimon/utils/ZOrderByteUtils.java
paimon-core/src/test/java/org/apache/paimon/utils/TestZOrderByteUtil.java
from http://iceberg.apache.org/ version 1.3.0

paimon-hive/paimon-hive-common/src/test/resources/hive-schema-3.1.0.derby.sql
Expand Down
21 changes: 21 additions & 0 deletions docs/content/concepts/append-only-table.md
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,27 @@ behavior is exactly the same as [Append For Qeueue]({{< ref "#compaction" >}}).
The auto compaction is only supported in Flink engine streaming mode. You can also start a compaction job in flink by flink action in paimon
and disable all the other compaction by set `write-only`.

### Sort Compact

The data in a per-partition out of order will lead a slow select, compaction may slow down the inserting. It is a good choice for you to set
write-only for inserting job, and after per-partition data done, trigger a partition `Sort Compact` action.

You can trigger action by shell script:
```shell
<FLINK_HOME>/bin/flink run \
/path/to/paimon-flink-action-0.5-SNAPSHOT.jar \
compact \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
--table <tableName> \
--order-strategy <orderType> \
--order-by <col1,col2,...>
```

{{< generated/sort-compact >}}

Other config is the same as [Compact Table]({{< ref "concepts/file-operations#compact-table" >}})

### Streaming Source

Unaware-bucket mode append-only table supported streaming read and write, but no longer guarantee order anymore. You cannot regard it
Expand Down
55 changes: 55 additions & 0 deletions docs/layouts/shortcodes/generated/sort-compact.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
{{/*
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
*/}}
{{ $ref := ref . "maintenance/configurations.md" }}
<!--
~ Licensed to the Apache Software Foundation (ASF) under one
~ or more contributor license agreements. See the NOTICE file
~ distributed with this work for additional information
~ regarding copyright ownership. The ASF licenses this file
~ to you under the Apache License, Version 2.0 (the
~ "License"); you may not use this file except in compliance
~ with the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->

<table class="configuration table table-bordered">
<thead>
<tr>
<th class="text-left" style="width: 15%">Configuration</th>
<th class="text-left" style="width: 85%">Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><h5>--order-strategy</h5></td>
<td>the order strategy now only support zorder. For example: --order-strategy zorder</td>
</tr>
<tr>
<td><h5>--order-by</h5></td>
<td>Specify the order columns. For example: --order-by col0, col1</td>
</tr>
</tbody>
</table>
Loading

0 comments on commit 0dbcff6

Please sign in to comment.