Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scale: Refurbish section, and absorb relevant tutorials from community forum #150

Merged
merged 8 commits into from
Nov 29, 2024
Merged

Scale: Refurbish section, and absorb relevant tutorials from community forum #150

merged 8 commits into from
Nov 29, 2024

Conversation

amotl
Copy link
Member

@amotl amotl commented Nov 27, 2024 •
edited
Loading

About

Rework the whole section about scaling CrateDB Clusters. Feedback is very much welcome.

Preview

Current: https://cratedb.com/docs/guide/admin/clustering/scale-up-down.html
Next: https://cratedb-guide--150.org.readthedocs.build/admin/clustering/scale/

Details

The patch absorbs those tutorials previously published on the community forum. Thanks for your excellent work. 💯 1

References

Footnotes

  1. I will add a bit of polishing to them on behalf of a subsequent iteration, if you don't mind. ↩

@amotl amotl mentioned this pull request Nov 27, 2024
Closed
amotl
amotl commented Nov 27, 2024
View reviewed changes
docs/admin/clustering/scale/index.md Outdated
Comment on lines 109 to 128
## Advices

Please apply best-practice operation guidelines when managing your database
cluster.

:::{tip}
:class: font-larger

For safely adding or decommission nodes, it is a good idea to
add and remove only one node at a time.
:::

:::{caution}
:class: font-larger

When removing nodes from a cluster, you must be conscious about maintaining a
quorum of nodes in the cluster as a whole, even if some nodes would not have
any data.
:::
Copy link
Member Author

@amotl amotl Nov 27, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's your suggestions, quickly slapped into the patch, @henrikingo. Please advise about any improvements.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I think for the last sentence, it's actually not about just removing nodes. Restarting or shutting down a node would do it too. Other than that this is really good!

Copy link
Member Author

@amotl amotl Nov 28, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Do you have a suggestion for a good wording that includes all relevant details concisely?

Copy link
Contributor

@henrikingo henrikingo Nov 28, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When restarting, shutting down or removing nodes from a cluster, using the DECOMMISION command... you must be conscious about maintaining a quorum of nodes up and connected to the cluster as a whole, even if some nodes would no longer have any data.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much. I've updated the patch using your suggestion.

@amotl amotl marked this pull request as ready for review November 27, 2024 19:30
@amotl amotl changed the title Scale: Refurbish section Scale: Refurbish section, and absorb relevant tutorials from community forum Nov 27, 2024
@amotl amotl mentioned this pull request Nov 27, 2024
Open
hammerhead
hammerhead reviewed Nov 28, 2024
View reviewed changes
docs/admin/clustering/scale/demand.md Outdated

We will be able to identify the new nodes by using a custom attribute (`node.attr.storage=temporarynodes`) (see further down for details on how to configure this), so the first step is to configure the existing partitions so that they do not consider the new nodes as suitable targets for shard allocation.

In CrateDB 5.1.2 or higher we can achieve this with:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't tested it yet, but since 5.9.0, it shouldn't be needed anymore to change allocation for every single table, but default allocation rules can be set on the cluster level:

Added support to override routing.allocation.* cluster settings with a routing.allocation.* table setting. This can be used to define the default routing behavior for all tables with a cluster setting and reroute individual tables by assigning the table setting using ALTER TABLE SET.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. So, let's add this as an additional information bit at this very spot, relating to "CrateDB 5.9.0 and higher"? Do you have different suggestions? Please advise.

Copy link
Member Author

@amotl amotl Nov 28, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi. 1e60c06 adds a relevant note, see comment below about possibly refining it. Thanks!

hammerhead
hammerhead reviewed Nov 28, 2024
View reviewed changes
docs/admin/clustering/scale/expand.md Outdated Show resolved Hide resolved
docs/admin/clustering/scale/auto.md Outdated Show resolved Hide resolved
docs/admin/clustering/scale/auto.md Outdated Show resolved Hide resolved
docs/admin/clustering/scale/auto.md Outdated Show resolved Hide resolved
@hammerhead @amotl
Scale: Inform about CrateDB 5.9.0's cluster-wide allocation rules
1e60c06 
In CrateDB 5.9.0 or higher, you don't need to change shard allocations for each
individual table any longer. Instead, default allocation rules can be set on
the cluster level:

> [CrateDB 5.9.0] added support to override `routing.allocation.*` cluster
> settings with a `routing.allocation.*` table setting. This can be used to
> define the default routing behavior for all tables with a cluster setting
> and reroute individual tables by assigning the table setting using `ALTER
> TABLE SET`.
amotl
amotl commented Nov 28, 2024
View reviewed changes
docs/admin/clustering/scale/demand.md
Comment on lines +51 to +66
In CrateDB 5.1.2 or higher, you can achieve this with:
```sql
/* this applies the setting to all existing partitions and new partitions */
ALTER TABLE test SET ("routing.allocation.exclude.storage" = 'temporarynodes');

/* then we run this other command so that the setting does not apply to new partitions */
ALTER TABLE ONLY test RESET ("routing.allocation.exclude.storage");
```
In CrateDB 5.9.0 or higher, you don't need to change shard allocations for each
individual table any longer. Instead, default allocation rules can be set on
the cluster level:
> [CrateDB 5.9.0] added support to override `routing.allocation.*` cluster
> settings with a `routing.allocation.*` table setting. This can be used to
> define the default routing behavior for all tables with a cluster setting
> and reroute individual tables by assigning the table setting using `ALTER
> TABLE SET`.
Copy link
Member Author

@amotl amotl Nov 28, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hammerhead suggested to educate readers about the improvements in CrateDB 5.9.0. Thanks!

Is it sensible to come up with example SQL statements here, similar like the statements above, to emphasize and clarify how a corresponding procedure would work?

NB: With new design elements in the documentation, we have a few more possibilites to convey information in a denser way, or optionally hide it from the main reading flow by using collapsibles. So, don't hesitate to increase the volume of documentation when relaying suggestions. I will try to give them a reasonable layout and formatting. Thanks!

Copy link
Member Author

@amotl amotl Nov 28, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you are fine with the notice, it will be all good. In this case, please acknowledge.

Otherwise, if you have a few examples at hand, and think adding them would be good, just slap them into a comment, and I will add them to the document at this spot.

henrikingo
henrikingo approved these changes Nov 28, 2024
View reviewed changes
Copy link
Contributor

@henrikingo henrikingo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for not approving earlier today, was interrupted

@amotl
Copy link
Member Author

amotl commented Nov 28, 2024 •
edited
Loading

No worries, and thanks. Let's merge the patch now, in order to serve the outcome to our readers in the spirit of release early and often?

@hammerhead: Please tell us about any improvements to the section you've suggested, also in retrospective. Thanks!

@amotl amotl merged commit 7aac0f4 into main Nov 29, 2024
3 checks passed
@amotl amotl deleted the amo/improve-scaling branch November 29, 2024 14:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@henrikingo henrikingo henrikingo approved these changes

@hammerhead hammerhead hammerhead approved these changes

@wierdvanderhaar wierdvanderhaar Awaiting requested review from wierdvanderhaar

@hlcianfagna hlcianfagna Awaiting requested review from hlcianfagna

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@amotl @hammerhead @henrikingo @hlcianfagna @wierdvanderhaar