RLink topologies and testing

[DZabavchik]

I was working on a PR that fixes some issues with RLinks (errors, lost and zombie registrations/subs, prefix/wildcard support).
I realized that I was targeting one topology (full mesh with reciprocal local forwarding from all to all)

Need some guidance on testing: Which topologies are realistic and should be tested?
I tried to codify topologies so configurations can be generated from simple spec. So here is the write up.

RLinks

Nomenclature
- 1..9, A..F - nodes with RLinks
- U,V,W,X,Y,Z - nodes without RLinks
- l - local
- r - remote
- lr - local and remote (both)

Node specification N(<target>[l | r | lr]), where N is the node number, target is the target node number, and l | r | lr - Local/Remote/Both

.. note::

Local (RLinkLocalSession) have authrole `trusted` and can see all registrations and events from
other all other sessions including remote (RLinkRemoteSession) sessions.

Remote (RLinkRemoteSession) have authrole `rlink` and can only see registrations local to the router,
but not from other rlinks. This is not the case for subscriptions:, remote session can see all events from
all sessions which results in full mesh subscriptions

Topologies

Two routers

• 1[Xl],X - simple node with RLink to X, forwarding local invocations/events to X
• 1[Xr],X - simple node with RLink to X, forwarding forwarding remote invocations/events from X
• 1[Xlr],X - bi-directional RLink to X

Chain

• 1[2l],2[3l],3[Xl],X - chain of nodes with RLinks to the next node, forwarding local invocations/events to the next node
• 1[2r],2[3r],3[Xr],X - chain of nodes with RLinks to the next node, forwarding remote invocations/events from the next node
• 1[2lr],2[3lr],3[Xlr],X - chain of nodes with bi-directional RLinks to the next node

Chain with reciprocal links

• 1[2l],2(1l,3l),3(2l,4l),4(3l) - chain of nodes with RLinks to the next node, forwarding local invocations/events to the next node, and reciprocal links to the previous node
• 1[2r],2(1r,3r),3(2r,4r),4(3r) - chain of nodes with RLinks to the next node, forwarding remote invocations/events from the next node, and reciprocal links to the previous node
• 1[2lr],2(1lr,3lr),3(2lr,4lr),4(3lr) - [INVALID] Oversubscribed chain of nodes with bi-directional RLinks to the next node, and reciprocal links to the previous node

Ring

• 1[2l],2[3l],3[4l],4[1l] - ring of nodes with RLinks to the next node, forwarding local invocations/events to the next node
• 1[2r],2[3r],3[4r],4[1r] - ring of nodes with RLinks to the next node, forwarding remote invocations/events from the next node
• 1[2lr],2[3lr],3[4lr],4[1lr] - ring of nodes with bi-directional RLinks to the next node

Ring with reciprocal links

• 1[2l],2(1l,3l),3(2l,4l),4(3l,1l) - ring of nodes with RLinks to the next node, forwarding local invocations/events to the next node, and reciprocal links to the previous node
• 1[2r],2(1r,3r),3(2r,4r),4(3r,1r) - ring of nodes with RLinks to the next node, forwarding remote invocations/events from the next node, and reciprocal links to the previous node
• 1[2lr],2(1lr,3lr),3(2lr,4lr),4(3lr,1lr) - [INVALID] Oversubscribed ring of nodes with bi-directional RLinks to the next node, and reciprocal links to the previous node

Star

• 1[Xl],2[Xl],3[Xl],4[Xl],X - star (inward) of nodes with RLinks to the central node, forwarding local invocations/events to the central node
• 1[Xr],2[Xr],3[Xr],4[Xr],X - star (inward) of nodes with RLinks to the central node, forwarding remote invocations/events from the central node
• 1[Xlr],2[Xlr],3[Xlr],4[Xlr],X - star (inward) of nodes with bi-directional RLinks to the central node
• 1(Xl, Yl, Zl),X,Y,Z - star (outward) of central node with RLinks to the outward nodes, forwarding local invocations/events to the outward nodes
• 1(Xr, Yr, Zr),X,Y,Z - star (outward) of central node with RLinks to the outward nodes, forwarding remote invocations/events from the outward nodes
• 1(Xlr, Ylr, Zlr),X,Y,Z - star (outward) of central node with bi-directional RLinks to the outward nodes

Star with reciprocal links

• 1[Cl],2[Cl],3[Cl],4[Cl],C(1l,2l,3l,4l) - star of nodes with RLinks to the central node, forwarding local invocations/events to the central node, and reciprocal links from the central node
• 1[Cr],2[Cr],3[Cr],4[Cr],C(1r,2r,3r,4r) - star of nodes with RLinks to the central node, forwarding remote invocations/events from the central node, and reciprocal links from the central node

Mesh

• 1[2l,3l,4l],2[1l,3l,4l],3[1l,2l,4l],4[1l,2l,3l] - mesh of nodes with RLinks to all other nodes and reciprocal links (all forwarding local invocations/events)
• 1[2r,3r,4r],2[1r,3r,4r],3[1r,2r,4r],4[1r,2r,3r] - mesh of nodes with RLinks to all other nodes and reciprocal links (all forwarding remote invocations/events)
• 1[2lr,3lr,4lr],2[1lr,3lr,4lr],3[1lr,2lr,4lr],4[1lr,2lr,3lr] - [OVERKILL/INVALID] mesh of nodes with bi-directional RLinks to all other nodes and reciprocal links

[oberstet]

I realized that I was targeting one topology (full mesh with reciprocal local forwarding from all to all)
Need some guidance on testing: Which topologies are realistic and should be tested?
[emphasis added by me]

++1! oh yes, I couldn't agree more!

what I mean is: to complete and to make rlink robust and ready for large scale production use, this is absolutely the approach we have to take.

we wouldn't be able to hash out bugs without regressions, and we wouldn't really increase confidence in correct working without. rlinks are by far the most complex piece of crossbar. it is significant LOCs, but that's not even the major problem, it is the complexity of what should happen (the routing semantics), and the complexity of features that rlinks build upon (e.g. meta API, router embedded client session, etc)

I tried to codify topologies so configurations can be generated from simple spec. So here is the write up.

++1! again, I very much agree, this is a very good approach. for 2 reasons (in my mind):

a) precise communication: and this again is essential, given the complexity and flexibility that ultimately will result if we can complete it)

b) functional test automation: if we can nail a DSL for describing topologies and node networks, we can add automation on top, which would create node networks on the fly, and fully integrated into CI/CD. and only with full (functional) test automation all of this is sustainable. and we will need that to survive without getting crazy;) there are some functional test setups and such here in the repo, but it is not currently active .. I got bogged down by the dependency dark hole. anyways ..

This is all fantastic!

rgd topologies: I will draw some pics of the most important topologies, and of some topologies that mark the ends/borders of the intended target design space (how far rlinks would ultimately carry), and also annotate/match your DSL to that! I'll come back later;)

[DZabavchik]

Reliance on meta API and embedded client sessions is exactly where I'm at right now. Here are some observations:

• Remote sessions (on remote router) get role rlink from principal role attribute. Router relies on that rlink role to prevent advertisement of new registration via wamp.registration.on_create (blocks propagation)
• Local sessions of the bridge are hardcoded to trusted, so whatever is registered on the remote is registered locally by trusted, which in turn allows further propagation to other remote sessions

This allows for chaining, but only in the direction of forwarding FROM remote. (forward_remote_invocations=True). Not the case in opposite direction.

There are several problems though:

1. There is disparity in registration visibility between live registrations (received via wamp.registration.on_create and list of registrations obtained from service via wamp.registration.list - this results in all sorts of issues:
   Disconnects and re-joins expose registrations that this RLink is not supposed to see -> propagation -> zombie registrations -> problems re-registering actual reconnecting proper client registrations
2. It is not possible to change direction of propagation of registrations or blocking propagation of RLink registrations

I was able to get clean logs (with clients pounding on startup, nodes dropping and reconnecting, clients rapidly jumping nodes and reregistering same URIs in a full mesh 1[2l,3l,4l],2[1l,3l,4l],3[1l,2l,4l],4[1l,2l,3l] by making all RLink sessions be in rlink role.

If we want to support chains, stars and meshes we need to have control over direction of propagation (role of RLinkLocalSession)

Notice this is only for registrations. Broker doesn't check if subscription came from RLink before publishing wamp.subscription.on_create, it relies on forwarded_for chain to prevent circular subscriptions.
-> So in a 4 node mesh it ends up with 12 RLink subscriptions (all to all others).
-> When session that created a subscription leaves, broker removes that observation, but there are 3 others (from RLinks)
-> Broker doesn't check if they are from RLinks (soft subs), number of observers is >0
-> Broker never publishes wamp.subscription.on_delete
-> RLink subscriptions stay

Please standby for PR.

[oberstet]

thanks a lot for your analysis and thoughts! IMO this is the right track / way to think about this stuff, and hash out the remaining issues to make rlinks complete.

just a few quick/preliminary comments, I'm working on the topology figures.

• forward_remote_invocations: great you've found that one;) yes, this (and similar) knobs are important
• "disparity in registration visibility": making the full meta API work in the presence of a meshed router network has definitely work still to do. e.g. what you noticed with wamp.registration.list: this needs to be transformed into a "distributed RPC" querying remote nodes - because even though we reflect registrations on a per node/URI basis, we do not reflect the complete set of client registration to all nodes (which obviously would not scale)
• "If we want to support chains, stars and meshes ": yes, we do want, or I should say, that was my plan with the design. crucially, I'd add "trees" as a topology pattern .. but I'll have that in the figures. fwiw, the most important topology I'm interested short term is: trees of edge routers connected to a full mesh in the cloud.
• beyond the topology in the last point, my mid-term goal is: allow to interconnect such trees-to-full-mesh network operated by some OP (node administrator) with similar network of other OPs
• and the longer term goal is: keep that cross OP full-mesh interconnects, but have edge nodes or edge trees operated by different OPs
• and the ultimate goal is: build on the last topology (cross OP full-meshes with edge nodes operated by different OPs), but have application payloads E2E encrypted - so that edge node OPs do not have to trust mesh node OPs (not trust other than for "general availability" ... but they shouldn't need to worry about mesh OPs selective availability .. that is dropping of individual events/calls based on app payloads .. as that payload is E2E encrypted)
• "So in a 4 node mesh it ends up with 12 RLink subscriptions (all to all others).": yes, if there is at least 1 subscriber (an actual client) connected and subscribed on each of the mesh nodes. if so, the number of RLink subscriptions will scale quadratically with number of mesh nodes - BUT is independent of total subscribers

[oberstet]

alright, I'll split up my figures, step by step;)

to start, rgd naming of owners / operators of router nodes and clients, my proposal would be to follow the established names (and respective first letters to abbreviate) from the cryptography community described e.g. here https://en.wikipedia.org/wiki/Alice_and_Bob#Cast_of_characters

what are your assumptions in the examples you've given? who is operator of what, or is the assumption that all router nodes are operated by one entity. what about clients?

mapping to WAMP:

[oberstet]

sorry, took a while as I was busy with other stuff. I've boiled down the topologies or elements thereof to target for rlinks and for rlink testing to the most important ones (in my eyes). @DZabavchik pls let me know what you think, let's discuss!

---

---

---

---

---

---

[DZabavchik]

In your topologies what does the arrow mean?. There is some information missing from topologies. Please standby, I'll draw and add what I think is missing.

Assuming arrow means Forward Local (Invocation/event). In that caseI'm absolutely sure about TP1, TP2, TP4. I need to test TP3 and TP5 (that's where chaining or forwarding of rlinks by rlinks is used).

Full mesh with all nodes forwarding to all others chaining is not necessary. This is achieved by assigning rlink role to both remote and local sessions. I used Full Mesh of 4.

Testing full mesh invocation (with orce_reregister: true)

C1 Registers test.rereg on R1
RLinkLocalSession] created forwarding registration: me=RLinkLocalSession(R1_to_R4 
RLinkLocalSession] created forwarding registration: me=RLinkLocalSession(R1_to_R3 
RLinkLocalSession] created forwarding registration: me=RLinkLocalSession(R1_to_R2 
C3 invokes test.rereg on R3 -> C1 receives invokation

C2 Registers test.rereg on R2
RLinkRemoteSession] Received UNREGISTERED : RLinkRemoteSession(R1_to_R2 
RLinkLocalSession] created forwarding registration: me=RLinkLocalSession(R2_to_R4 
RLinkLocalSession] created forwarding registration: me=RLinkLocalSession(R2_to_R3 
RLinkRemoteSession] Received UNREGISTERED : RLinkRemoteSession(R1_to_R4 
RLinkRemoteSession] Received UNREGISTERED : RLinkRemoteSession(R1_to_R3 
RLinkLocalSession] created forwarding registration: me=RLinkLocalSession(R2_to_R1 

C3 invokes test.rereg on R3 -> C2 receives invocation

C2 disconnects
RLinkLocalSession] Deleting registration of test.rereg on RLinkLocalSession(R2_to_R1 
RLinkLocalSession] Deleting registration of test.rereg on RLinkLocalSession(R2_to_R4 
RLinkLocalSession] Deleting registration of test.rereg on RLinkLocalSession(R2_to_R3 

URI is no longer registered on any routers
No errors in logs
PASS

Testing full mesh subs (single path routing)

C1 subscribes to test.pub.static on R1

created forwarding subscription or test.pub.static: me=RLinkLocalSession(R1_to_R3
created forwarding subscription or test.pub.static: me=RLinkLocalSession(R1_to_R4
created forwarding subscription or test.pub.static: me=RLinkLocalSession(R1_to_R2
  -- notice that only direct forwarding subscriptions are registered on remotes, it does not produce an avalanche of registrations, 

C3 on R3 publishes an event
RLink forward-published event from remote to local  ... (forward_for=[{'session': 4500421128452903, 'authid': 'RECF-P7TY-FFVS-SEG9-E97S-A3QT'
C1 receives the event


C2 subscribes to test.pub.static on R2
created forwarding subscription or test.pub.static: me=RLinkLocalSession(R2_to_R3
created forwarding subscription or test.pub.static: me=RLinkLocalSession(R2_to_R4
created forwarding subscription or test.pub.static: me=RLinkLocalSession(R2_to_R1

C3 on R3 publishes an event
RLink forward-published event from remote to local ... forward_for=[{'session': 4500421128452903, 'authid': 'RECF-P7TY-FFVS-SEG9-E97S-A3QT'
RLink forward-published event from remote to local ... forward_for=[{'session': 4500421128452903, 'authid': 'RECF-P7TY-FFVS-SEG9-E97S-A3QT'
C1 & C2 receive the event 


C1 unsubscribes/disconnects
unsubscribed from test.pub.static on RLinkRemoteSession(R1_to_R3
unsubscribed from test.pub.static on RLinkRemoteSession(R1_to_R2
unsubscribed from test.pub.static on RLinkRemoteSession(R1_to_R4


C3 on R3 publishes an event
RLink forward-published event from remote to local ... forward_for=[{'session': 4500421128452903, 'authid': 'RECF-P7TY-FFVS-SEG9-E97S-A3QT'

C2 unsubscribes/disconnects
unsubscribed from test.pub.static on RLinkRemoteSession(R2_to_R1
unsubscribed from test.pub.static on RLinkRemoteSession(R2_to_R3
unsubscribed from test.pub.static on RLinkRemoteSession(R2_to_R4

----
No mangling subscriptions are left on any of the routers
No errors in logs
PASS

[oberstet]

In your topologies what does the arrow mean?. There is some information missing from topologies. Please standby, I'll draw and add what I think is missing.

good question. also, will wait for your complete feedback, but here are some aspects I wanted to communicate:

the arrow direction as used in my diagram was only intended to communicate the direction of establishing the rlink at the network level. thus, the destination has a listening IP/port, whereas the source does not necessarily have one.

now, that direction of network connection establishment is just a necessary precondition, but it does not yet presume anything about the forward_remote_invocations and similar flags - which of course ultimately determine what happens at the WAMP level independent of the network connectivity direction

IOW: there are definitely crucial things not yet communicated in "that one arrow" alone.

we need to discuss ... I was starting from "ground up" with most important elements first ... in my eyes, that would be "network connectivity direction" and "networks / ownership"

[oberstet]

note: also filed this (the figures) here wamp-proto/wamp-proto#503

[oberstet]

alright, I've translated the architecture that is used in a real-world project I am currently involved. this is the last one, promised;)

however, it is the one I am actually using in an application and hence the topology I'm really interested in. if I could test that automatically, that would be awesome of course. and it basically uses all the elements from above.

it consists of

• 3 networks and operators/owners
• 12 router nodes in total
• different kind of clients connected to basically/potentially any of the router nodes

this diagram only shows router nodes, no clients, no network and cross-network link details like firewalls/NATs to keep it simple.

---

[DZabavchik]

Thank you, this is perfect. We should use this nomenclature router ID formatting with network ID
R- Router
A-Z - Network
1-N - Router index within network

RA3 - Alice network Router 3

[oberstet]

yes, exactly. having a "standard" ID formatting for network/owner and router IDs. and attackers. so to be precise, what I propose:

• R Router and C Client (peer type - first character of ID)
• A-F Network/Owner and G, H, N, N, T Attackers (entity - second character of ID)
• 1, 2, ..., n Router (or Client) index within network / owner (peer index - third character of ID)

thus, it is not "A-Z", because "G .." is already used for an attacker ID:

[DZabavchik]

Here is my version of TP3 + Legend

[oberstet]

awesome! I need to think about this ... will reply later

[DZabavchik]

To illustrate the difference between solid arrows (forwarding from others), consider these diagrams of a mesh (TP4)

Yeah, I need a break too. Have a great weekend.