Multi-threaded publishing for router?

[ayuk23]

Hey! I'm taking over a legacy application at my company that utilizes crossbar. We have a handful of publisher apps that publish data over WAMP and then a bunch of users that selectively subscribe from their browsers via our websocket router to that data.

We've been experiencing performance issues and someone on the ops team identified a bottleneck. It seems like there's a single thread that publishes data to all of our clients. He recommended that we increase the thread count if possible. I've been navigating the documentation here but haven't found any configuration to do so. Is there anyway to achieve this in crossbar?

Also worth mentioning that we are on v21.1.1

Thanks in advance!

[oberstet]

Yes, Crossbar.io (using router-to-router links, and proxy and router workers) scales with the number of CPU cores, e.g. a number I use for orientation is, 50k fully routed WAMP RPCs (each consisting of 4 WAMP messages on the wire) per CPU core using 2-8 GB RAM per CPU core ...

[ayuk23]

Level 1 is probably sufficient for our use case.

https://github.com/crossbario/crossbar/blob/master/docs/Router-Configuration.rst

Is the necessary configuration covered here? We have a single transport but would want multiple CPUs cores publishing on that transport?

[oberstet]

it's not documented.

here is an example using pretty raw means, just one example with one specific approach

https://github.com/crossbario/crossbar-examples/tree/master/rlinks

[ayuk23]

I'm sorry but I think this example that you gave is for Level 2 and not Level 1.

For our use-case it's sufficient to scale up an individual node. Whereas this link describes "a fully managed network of crossbar nodes all managed by a single master node."

Any idea how we can configure our router.json to scale up an individual node?

[ayuk23]

https://github.com/crossbario/crossbar-examples/tree/master/benchmark/websocket

I believe it's actually this example. Thanks again for your help!

[ayuk23]

@oberstet

So, I tried following the example here - https://github.com/crossbario/crossbar-examples/tree/master/benchmark/websocket. Our config looks something like:

{
  "version": 2,
  "workers": [
    {
      "type": "router",
      "realms": [
        {
          "name": "realm1",
          "roles": [
            {
              "name": "authorizer",
              "permissions": [
                {
                  "uri": "*",
                  "allow": {
                    "register": true,
                    "call": true,
                    "publish": true,
                    "subscribe": true
                  },
                  "disclose": {
                    "caller": true
                  }
                }
              ]
            },
            {
              "name": "anonymous",
              "authorizer": "hub.authorizer"
            }
          ]
        }
      ],
      "transports": [
        {
          "type": "websocket",
          "endpoint": {
            "type": "tcp",
            "port": 8083,
            "shared": true,
            "backlog": 1024
          }
        }
      ],
      "components": [
        {
          "type": "class",
          "classname": "wamp_authorizer.HubAuthorizer",
          "realm": "realm1",
          "role": "authorizer"
        }
      ]
    },
...
]

With this first worker blob essentially duplicated again two create two workers. Our crossbar routers spin up just fine with no issues. But when our UI client connects to the websocket and hits the authorization rpc, our autobahn connection call gives us a 'wamp.error.no_such_procedure' error whenever it connects to the second worker. I can see in the logs that the second worker is receiving the rpc call and returning a response but it is lost in the UI.

The way our system is setup is that we have a variety of publisher apps publishing data via websocket into our crossbar router. And then Ui clients selectively subscribe to that data (as well as make a handful of different RPCs).

It also seems like with this multi-worker router setup, each router isn't getting the full set of data. Still being newcomer to crossbar, this seems to make sense since the two router workers would not be sharing realms (even though they are the same name).

Going back to your previous reply about using RLinks - I'm assuming this is the way to forward events from router to router. I recreated the configuration for the rlink/smp_setup example here - https://github.com/crossbario/crossbar-examples/blob/master/rlinks/smp_setup/edge1/.crossbar/config.json

    {
      "type": "router",
      "realms": [
        {
          "name": "realm1",
          "roles": [
            {
              "name": "authorizer",
              "permissions": [
                {
                  "uri": "*",
                  "allow": {
                    "register": true,
                    "call": true,
                    "publish": true,
                    "subscribe": true
                  },
                  "disclose": {
                    "caller": true
                  }
                }
              ]
            },
            {
              "name": "anonymous",
              "authorizer": "hub.authorizer"
            },
            {
              "name": "router2router",
              "permissions": [
                {
                  "uri": "",
                  "match": "prefix",
                  "allow": {
                    "call": true,
                    "register": true,
                    "publish": true,
                    "subscribe": true
                  },
                  "disclose": {
                    "caller": false,
                    "publisher": false
                  },
                  "cache": true
                }
              ]
            }
          ],
          "rlinks": [
            {
              "id": "rlink_1_2",
              "realm": "realm1",
              "transport": {
                "type": "rawsocket",
                "endpoint": {
                  "type": "unix",
                  "path": "edge2.sock"
                },
                "url": "rs://localhost",
                "serializer": "cbor"
              },
              "forward_local_events": false,
              "forward_remote_events": true
            }
          ]
        }
      ],
      "transports": [
        {
          "type": "rawsocket",
          "endpoint": {
            "type": "unix",
            "path": "edge1.sock",
            "backlog": 1024
          },
          "serializers": ["cbor"],
          "auth": {
            "anonymous": {
              "type": "static",
              "role": "router2router"
            }
          }
        },
        {
          "type": "websocket",
          "endpoint": {
            "type": "tcp",
            "port": 8083,
            "shared": true
          }
        }
      ],
      "components": [
        {
          "type": "class",
          "classname": "wamp_authorizer.HubWampAuthorizer",
          "realm": "realm1",
          "role": "authorizer"
        }
      ]
    }
    ...

After doing this, I still observe the same error whenever connecting to the second worker. Checking the logs, I see a TransportLost exceptions when creating both rlinks.

  File "crossbar/worker/rlink.py", line 460, in onJoin
    yield self._setup_event_forwarding(remote)
  File "crossbar/worker/rlink.py", line 221, in _setup_event_forwarding
    yield on_subscription_create(sub_id, sub)
  File "twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "crossbar/worker/rlink.py", line 177, in on_subscription_create
    sub = yield other.subscribe(on_event, uri, options=SubscribeOptions(details=True))
  File "autobahn/wamp/protocol.py", line 1490, in subscribe
    raise exception.TransportLost()

Just curious if anything is obviously incorrect about my setup. We are on an earlier version of crossbar (v21.1.1). From the changelog, it seems like there have been fixes to RLinks in later versions. It is unfortunately a pain for us to upgrade to a new version since our legacy build system currently only supports Python 3.6.

If this is something that can only work in a future release of crossbar, then our team can discuss making the upgrade. But we would like some confidence first that this works.

Regardless, we appreciate your time thus far.

[oberstet]

lemme comment, just quickly ..

We are on an earlier version of crossbar (v21.1.1). From the changelog, it seems like there have been fixes to RLinks in later versions. It is unfortunately a pain for us to upgrade to a new version since our legacy build system currently only supports Python 3.6.

Yes indeed, there has been quite some development and fixing happening rgd rlinks, r2r and such in Crossbar.io since v21.1.1, and based on a pretty advanced and extensive setup for a customer project to test and verify, so I am confident to say: if you want to use this stuff, you should use the very latest Crossbar.io tagged release, which is v23.1.2 currently - need to push a new release summing up last months changes.

You don't need to and should not build Crossbar.io, but you can just use the official Docker images published, which includes everything, and runs on PyPy.

our team can discuss making the upgrade

Seems like a process problem: your team should not be required to "make" or do anything just to follow Crossbar.io stable releases

Maybe update 1 line somewhere which says which release version you follow. And then "make pull / update" to trigger the rollout.

[ayuk23]

Maybe update 1 line somewhere which says which release version you follow. And then "make pull / update" to trigger the rollout.

We do have 1 line that we can update but doing that causes our build to fail because min version is greater than Python 3.6. We could do the upgrade for only the component running crossbar. That would be a much smaller lift.

Will using rlinks solve the problems we're experiencing? Both with the authorization not working and not receiving the full set of data within your session?

[oberstet]

causes our build to fail

I would recommend to throw away your build and use the official latest Docker image.

Will using rlinks solve the problems we're experiencing?

Maybe. Try it, then you know.