Skip to content

Commit

Permalink
[FLINK-36198][network] Change document related to hybrid shuffle
Browse files Browse the repository at this point in the history
  • Loading branch information
reswqa committed Sep 3, 2024
1 parent ae1d7c1 commit 13cd826
Show file tree
Hide file tree
Showing 2 changed files with 0 additions and 20 deletions.
10 changes: 0 additions & 10 deletions docs/content.zh/docs/ops/batch/batch_shuffle.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,16 +131,6 @@ These could be configured via [jobmanager.partition.hybrid.partition-data-consum

Hybrid shuffle supports to store the shuffle data to the remote storage. The remote storage path can be configured by [taskmanager.network.hybrid-shuffle.remote.path]({{< ref "docs/deployment/config" >}}#taskmanager-network-hybrid-shuffle-remote-path). This feature supports various remote storage systems, including OSS, HDFS, S3, etc. See [Flink Filesystem]({{< ref "docs/deployment/filesystems/overview" >}}) for more information about the Flink supported filesystems.

Note: The remote storage is only supported in the new hybrid shuffle mode. For more details about the new mode, please refer to the following [The New Mode And The Legacy Mode]({{< ref "docs/ops/batch/batch_shuffle#the-new-mode-and-the-legacy-mode" >}}).

### The New Mode And The Legacy Mode

We have refactored the architecture of Hybrid shuffle in Flink 1.18 (new mode), to resolve some existing issues. It has several advantages compared to the original architecture (legacy mode), such as requiring less network memory and supporting remote storage.

The new mode is enabled by default. To switch back to the legacy mode, set [taskmanager.network.hybrid-shuffle.enable-new-mode]({{< ref "docs/deployment/config" >}}#taskmanager-network-hybrid-shuffle-enable-new-mode) to `false`.

Note: The legacy mode is deprecated and can be removed in future releases.

### Limitations

Hybrid shuffle mode is still experimental and has some known limitations, which the Flink community is still working on eliminating.
Expand Down
10 changes: 0 additions & 10 deletions docs/content/docs/ops/batch/batch_shuffle.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,16 +131,6 @@ These could be configured via [jobmanager.partition.hybrid.partition-data-consum

Hybrid shuffle supports to store the shuffle data to the remote storage. The remote storage path can be configured by [taskmanager.network.hybrid-shuffle.remote.path]({{< ref "docs/deployment/config" >}}#taskmanager-network-hybrid-shuffle-remote-path). This feature supports various remote storage systems, including OSS, HDFS, S3, etc. See [Flink Filesystem]({{< ref "docs/deployment/filesystems/overview" >}}) for more information about the Flink supported filesystems.

Note: The remote storage is only supported in the new hybrid shuffle mode. For more details about the new mode, please refer to the following [The New Mode And The Legacy Mode]({{< ref "docs/ops/batch/batch_shuffle#the-new-mode-and-the-legacy-mode" >}}).

### The New Mode And The Legacy Mode

We have refactored the architecture of Hybrid shuffle in Flink 1.18 (new mode), to resolve some existing issues. It has several advantages compared to the original architecture (legacy mode), such as requiring less network memory and supporting remote storage.

The new mode is enabled by default. To switch back to the legacy mode, set [taskmanager.network.hybrid-shuffle.enable-new-mode]({{< ref "docs/deployment/config" >}}#taskmanager-network-hybrid-shuffle-enable-new-mode) to `false`.

Note: The legacy mode is deprecated and can be removed in future releases.

### Limitations

Hybrid shuffle mode is still experimental and has some known limitations, which the Flink community is still working on eliminating.
Expand Down

0 comments on commit 13cd826

Please sign in to comment.