Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Edits to Fix default setting and type for force_full_mesh. #6188 #6245

Merged
merged 3 commits into from
Nov 21, 2024
Merged

Edits to Fix default setting and type for force_full_mesh. #6188 #6245

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7cd95cf
Edits to Fix default setting and type for force_full_mesh. #6188
ebgitelman Nov 15, 2024
0da5d5d
Clean up phrasing
djw-m Nov 18, 2024
26a46fb
Update product_docs/docs/pgd/5.6/nodes/subscriber_only/optimizing-so.mdx
ebgitelman Nov 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 22 additions & 20 deletions product_docs/docs/pgd/5.6/nodes/subscriber_only/optimizing-so.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,73 +1,75 @@
---
title: Optimizing subscriber-only groups
navTitle: Optimizing
description: 'Optimizing subscriber-only groups in EDB Postgres Distributed'
description: 'Optimizing subscriber-only groups in EDB Postgres Distributed.'
---

With PGD 5.6 and later, it's possible to optimize the topology of [subscriber-only groups](overview).

In this optimized topology, a small number of fully active nodes, the write leaders of the data groups, replicate changes to the group leaders of subscriber-only groups, these group leaders then replicate changes to the other members of its subscriber-only group.
In this optimized topology, a small number of fully active nodes—the write leaders of the data groups—replicate changes to the group leaders of subscriber-only groups. These group leaders then replicate changes to the other members of its subscriber-only group.
djw-m marked this conversation as resolved.
Show resolved Hide resolved

## Requirements for the optimized topology

You can't enable this model if a cluster has any of the following:

* Data-nodes which are directly members of the top-level group.
* No data-node sub-groups.
* No data-node sub-groups with proxy routing enabled.
* Data nodes that are directly members of the top-level group
* No data-node subgroups
* No data-node subgroups with proxy routing enabled

If this is the case, the nodes in a subscriber only groups automatically revert to the full mesh topology.
If any of these are the case, the nodes in subscriber-only groups revert to the full mesh topology.

To get the benefit of the new SO group and node replication, you must have your data-nodes in subgroups, with proxy routing enabled on the sub groups.
To get the benefit of the new SO group and node replication, you must have your data nodes in subgroups, with proxy routing enabled on the subgroups.

## How the optimized topology works

For clusters using groups for their data nodes, subscriber only groups can use a more efficient model which uses a subscriber-only group leader, similar to a write leader in PGD Proxies.
For clusters using groups for their data nodes, subscriber-only groups can use a more efficient model. This model uses subscriber-only group leaders, similar to write leaders in PGD proxies.

Each subscriber-only group uses that group leader to replicate changes to other subscriber-only nodes in its group. The group leader acts as a replication proxy for incoming changes.

The write leader nodes in other non-subscriber-only groups replicate changes to the group leader of the subscriber-only group. Other nodes only replicate within their data groups, both within their own group and in other data groups.
The write leader nodes in data groups replicate changes to the group leaders of the subscriber-only groups. Other nodes in the data groups only replicate with nodes in their data group and with data nodes in other data groups. They do not directly replicate their changes to the subscriber-only groups.

## Subscriber-only group leaders

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the para before this section head, I wanted to check whether the second sentence needed some additional punctuation or if there was some error, as it seemed to contradict itself.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have made changes to make it clearer.

With PGD 5.6 and later, each subscriber-only group gets assigned a group leader of its own.
This is because subscriber-only groups don't have a group Raft consensus mechanism of their own.
Instead, the cluster’s top level group uses its Raft consensus mechanism to handle selection of each subscriber-only group’s group leader.
This group leader selection is on by default on PGD 5.6, regardless of the topology optimization settings.
Instead, the cluster’s top-level group uses its Raft consensus mechanism to handle selecting each subscriber-only group’s group leader.
This group leader selection is on by default in PGD 5.6, regardless of the topology optimization settings.

Group leaders in subscriber-only groups are regularly tested for connectivity and if unavailable, the voting nodes of top-level group select a new subscriber-only node from the subscriber-only group to become group leader.
The new group leader is then selected.
Group leaders in subscriber-only groups are regularly tested for connectivity and, if unavailable, the voting nodes of top-level group select a new subscriber-only node from the subscriber-only group to become group leader.
The new group leader is then selected.

With optimized technology turned off, this election has no effect on the replication topology. Without the optimized topology, all data nodes replicate changes to all other nodes in the cluster.

## Group leaders in the optimized topology

With the optimized topology enabled, only the subscriber-only group's group leader receives changes from other data group's write leaders in the cluster and it takes on the responsibility of replicating those changes to the other nodes in the subscriber-only group.
With the optimized topology enabled, only the subscriber-only group's group leader receives changes from other data groups' write leaders in the cluster. The group leader takes on the responsibility of replicating those changes to the other nodes in the subscriber-only group.

The other voting nodes choose the group leader from a subscriber-only group's nodes.
Once selected, the whole cluster becomes aware of the change and any data group's write leaders then replicate data only to this newly selected group leader node.
Once the group leader is selected, the whole cluster becomes aware of the change, and any data group's write leaders then replicate data only to this newly selected group leader node.
Other data nodes in the data groups don't replicate data to the subscriber-only group's nodes.

This approach avoids the explosion of active connections that can happen when there are large numbers of SO nodes and reduces the amount of replication traffic.

The Subscriber-Only node and group form the building block for PGD tree topologies.
The subscriber-only node and group form the building block for PGD tree topologies.

## Enabling the optimized model

By default, PGD 5.6 forces the full mesh topology. This means the optimization described here is off.
To enable the optimized topology, you must have your data-nodes in subgroups, with proxy routing enabled on the sub groups.
To enable the optimized topology, you must have your data nodes in subgroups, with proxy routing enabled on the subgroups.
You can then set the GUC [`bdr.force_full_mesh`](/pgd/5.6/reference/pgd-settings#bdrforce_full_mesh) to `off` to allow the optimization to be activated.
Note: This GUC needs to be set in the postgresql.conf file on each data node and each node restarted for the change to take effect.

If any requirements of the optimized topology aren't met, the nodes in a subscriber-only group automatically revert to the full mesh topology. When this happens, you'll find in the logs of the nodes in the cluster messages that explain why the optimization was not possible, such as:
!!! Note
This GUC needs to be set in the `postgresql.conf` file on each data node and each node restarted for the change to take effect.

If any requirements of the optimized topology aren't met, the nodes in a subscriber-only group revert to the full mesh topology. When this happens, you'll find in the logs of the nodes in the cluster messages why the optimization wasn't possible, such as:
djw-m marked this conversation as resolved.
Show resolved Hide resolved

When a data node is part of the top-level node group:

```plaintext
node: <nodename> is part of top-level nodegroup: <toplevelgroupname>: changing to full mesh".
```

and when a data group does not have proxy routing enabled:
When a data group doesn't have proxy routing enabled:

```plaintext
node: <nodename> is in nodegroup: <nodegroupname> that does not have proxy routing: changing to full mesh.
Expand Down