Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added node limit to P2P topology #1739

Closed

Conversation

michaeljfazio
Copy link
Contributor

This commit adds limit to the amount of gossips a node will store in memory at any one time. This allows nodes to be resilient against acquiring too many stale node profiles from the network.

This issue definitively fixes #1599 and makes the processing requirements of Jormungandr deterministic in relation to the total number of network profiles being gossiped in the network.

This is what we see before the change. The processing requirements are increasing relative to the number of profiles being processed into p2p overlays.

72666360-5539d380-3a4c-11ea-9904-d18f90c3aec2

Nodes will become so bogged down that they start to miss block production (see #1580). Eventually they fall too far behind the tip of the main chain, resulting in complete de-sync. Longer term, nodes crash completely due to expended resources.

Here is the result after this change with default max peer size set to 10000.

Screen Shot 2020-02-06 at 8 41 58 pm

In this case, my CPU usage never climbs above 2% and the node never de-syncs from the network.

A follow up to this issue will allow the maximum number of stored node profiles to be configured. Though 10000 as a default for the time being seems to be more than sufficient for most nodes.

This solution doesn't solve the underlying issue of nodes forever being gossiped on the network. To solve this will probably require a protocol level enhancement (i.e. including liveness timestamp in propagated gossip). However, the fix also does not detract from the poldercast specification. It simply accepts the resource limitations of the node.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Jormungandr consumes 100% CPU and crashes
5 participants