Is it possible to do stickiness based on the io cookie or custom http headers in NGINX?

[antonander]

Hello, all.

TLDR; Is it possible to do stickiness in NGINX based on the socket.io cookie (io) or a custom HTTP header? Or is this only possible with the IP?

I am currently looking at running an application on multiple nodes. We use NGINX and want to keep the support for polling as transport method.

The docs (https://socket.io/docs/v4/using-multiple-nodes#nginx-configuration) show an example of the configuration needed in NGINX to do the load-balancing with stickiness based on the ip_hash or hash $remote_addr consistent methods.

This works fine, I am looking, however, at being able to do this with the cookie set. In the docs related to cookies (https://socket.io/how-to/deal-with-cookies#cookie-based-sticky-session), it mentions:

This cookie can then be used for cookie-based sticky session, when scaling to multiple nodes (example with HAProxy here).

My attempt to do this with NGINX Open Source looks something like this:

http {
  server {
    listen 80;
    listen [::]:80;
    server_name _;

    location / {
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $host;

      proxy_pass http://nodes;

      # enable WebSockets
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
    }
  }

  upstream nodes {
  # also tried hash $cookie_io consistent;
    hash $cookie_io 
    server 127.0.0.1:3000;
    server 127.0.0.1:3001;
  }
}

This sort of works, I don't get as many 400 errors, but I still end up with some. It can stablish a successful connection after a few attempts, but it still seems odd. I don't see this happening when using ip_hash. I see the cookie is indeed included in the requests. This is the server setup:

  const socketsServer = createServer(server);
  
  const options = {
    origin: (_req, callback) => {
      callback(null, true);
    },
    credentials: true,
    pingInterval: 3000,
    serveClient: false,
    cookie: true,
    adapter: createAdapter(redis.getClient())
  }
  
  const io = new Server(socketsServer, options)

I also tried to add a custom header (https://socket.io/docs/v3/client-initialization/#extraheaders) and use that to do the load balancing:

http {
  server {
   ...
  }

  upstream nodes {
   # also tried hash $http_customHeader consistent;
    hash $http_customHeader
    server 127.0.0.1:3000;
    server 127.0.0.1:3001;
  }
}

But I got the same result as with $cookie_io

I'm trying to debug the issue, but I wonder, is this even supported? I see no indication or example on how to do this with NGINX.

Thanks in advance.

[darrachequesne]

Hi! Unfortunately, the cookie directive is only available with NGINX Plus.

Reference: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#sticky_cookie