UDP fragmentation

[softins]

https://github.com/corrados/jamulus/blob/017796919c814a74adcf916a6f200e7c157e58f1/ChangeLog#L19

Going down from 200 to 150 won't remove UDP fragmentation. In a Wireshark trace a couple of weeks ago, a list of 198 servers took 8 fragments, which when reassembled had a payload size of 6780 bytes. The fragments in that instance appeared to have an MTU of only 778 bytes, which is something over which neither end has control, as it may be due to an intermediate hop. Even if the path had the maximum possible MTU of 1500 bytes, it would still take around 4 or 5 fragments to send a list of 200 servers.

The only way really to solve it would be for the server also to listen on a TCP port (could use the same port number). A client wanting the server list could then connect to the central server on TCP, fetch the list and then close the connection. Fragmentation should no longer be a problem then.

For backward compatibility, a client could fall back to UDP if the TCP connection was refused, and the central server would still serve UDP requests for the server list if asked.

(Of course, it is only broken or misconfigured routers or computers that do not correctly handle fragmented UDP packets)

[corrados]

Wie will have multiple server lists in the near future. Each list size can even be further reduced to, e.g., 100 or less servers. This will solve the issue.

[corrados]

Users reported not long ago that NorthAmerica list worked but not the Default. At that time the NorthAmerica list already had about 100 entries so I assume that this will work.

[streaps]

IMHO protocol implementation issues shouldn't dictate the size of the server list and force a fragmentation into multiple central servers. This is really a bad design. Before I read this issue I already wondered (as a user who discovered Jamulus) why there is a limit of 200.

I don't know the the code and the protocol, so I don't know how hard or easy it is to change it (compatibility to older clients and servers,...). There is also QUIC as an alternative to TCP and plain UDP. QUIC also would make it possible to connect with a web client in the future.

[corrados]

This is really a bad design.

Well, it worked remarkably well the last 14 years ;-).

shouldn't dictate the size of the server list

The large server list does not only have UDP fragmentation issues but others as well. E.g. the client pings all the servers in the list every some seconds. This is a fundamental thing. It is much better to have multiple small lists.
Fortunately, pljones voluteers to host two more lists. So we will have four lists with the next Jamulus release (hopefully this weekend) and maybe even more in the future.

[streaps]

This is really a bad design.

Well, it worked remarkably well the last 14 years ;-).

I wonder why there is an open issue about it, if it still works well. ;)

[corrados]

Well, it seems you are a bit impatient ;-)

[lefty665]

I made this comment a couple of days ago on sourceforge but was redirected here.

Hi Guys, Help please. I'm in the US, and getting erratic connect lists. Sometimes it's full lists, sometimes partial lists, and most often no lists. Loading with --showallservers the servers with ping times, and not all have them, look a lot like the partial lists, when they show up at all. Most times it works the first time after a boot. Simply closing Jamulus and reopening it does not re-enable server lists. Once it quits giving lists it's done until a reboot. Older 8 core computer @4ghz, 16gb ram, but poky DSL, although far faster than spec minimums. Any suggestions, ports I should look at, etc are appreciated.

Other than connection issues I'm a fan, Jamulus has let me keep up with band members and musician friends while we're sequestered. Thank you.

ps. a fellow on sourceforge embraced the idea of turning off the stateful packet inspection portion of a users firewall and indicated he was going to add it to the Wiki Jamulus post. Doesn't seem like a great idea. https://sourceforge.net/p/llcon/discussion/533517/thread/0e9aa52428/?page=1

Thanks for your help.

[gilgongo]

@lefty665 The forthcoming update (3.5.4) implements a feature designed to help with the issue of seeing fewer servers than you should. This may or may not cure it, but it would be good if you could test it out. Watch out for the release announcement.

As to the firewall issue, I see they have replied to that, but essentially you will be safe without SPI unless you are doing something pretty non-standard with your network. In which case you would know whether you need SPI or not.

[corrados]

Loading with --showallservers the servers with ping times, and not all have them, look a lot like the partial lists, when they show up at all.

Do you always see the full list if you use --showallservers (ignoring that not all servers have a ping time)?

[gilgongo]

Do you always see the full list if you use --showallservers

That's certainly my own experience. Would be interesting to know about others.

[corrados]

That's certainly my own experience.

If this is the case, then we are in the wrong Github Issue here. Because if you see the full list with --showallservers, you are not affected by the UDP fragmentation issue (see the title of this Issue) since you can receive the complete list. It is then an issue with the ping messages which are suppressed.

[gilgongo]

Ah OK. I thought the two things were connected.

[lefty665]

Loading with --showallservers the servers with ping times, and not all have them, look a lot like the partial lists, when they show up at all.

Do you always see the full list if you use --showallservers (ignoring that not all servers have a ping time)?

No, it will show a full list once, occasionally twice, then a blank data list until it is rebooted. If I had to guess it would more likely be a ping issue. Jamulus apparently generates a lot of pings. Could it be overrunning a buffer or exceeding a threshold? But, that's just guesses from what I can see, not an informed opinion from getting my nose in the code. Thank you for your response, and for Jamulus.

[lefty665]

@lefty665 The forthcoming update (3.5.4) implements a feature designed to help with the issue of seeing fewer servers than you should. This may or may not cure it, but it would be good if you could test it out. Watch out for the release announcement.
As to the firewall issue, I see they have replied to that, but essentially you will be safe without SPI unless you are doing something pretty non-standard with your network. In which case you would know whether you need SPI or not.

I'll look for 3.54, thank you.

In reference to the firewall issue, I don't see that response, please be kind enough to direct me to it. - I found your response on sourceforge. My response is "horsefeathers". Your explanation of web threats is naïve at best. Having both hardware/router and software/os firewalls is entry level web security.

I do see from the wiki that recommending disabling SPI is now included. SPI is a fundamental feature of many better router firewalls. It is concerning that you would recommend disabling it to a user community that is often not well versed in web threats. In addition to potential damage to user computers your recommendation to trusting users creates potential liability for damages they may incur. I encourage you to remove that recommendation.

Tell me please, is Jamulus "doing something pretty non-standard with your network" traffic that would run afoul of SPI? Thanks. https://en.wikipedia.org/wiki/Stateful_firewall

[gilgongo]

SPI is a fundamental feature of many better router firewalls.

That discussion isn't relevant to this ticket. Please continue it on the forum thread.

[corrados]

either not sorting by ping time

If the mouse pointer is located over the table, the list is not sorted. So you have to move the mouse pointer away from the list so that it is getting sorted.

[lefty665]

Thanks, guess I've been too compulsive with the mouse, scrolling up and down the unsorted table cursing and looking for low ping times. ;-)

…

On 10/15/2020 11:33 AM, Volker Fischer wrote: either not sorting by ping time If the mouse pointer is located over the table, the list is not sorted. So you have to move the mouse pointer away from the list so that it is getting sorted. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#255 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APWAKF5EZJ6YNN4GJJSSNU3SK4I6JANCNFSM4NGY5CNA>.

-- This email has been checked for viruses by AVG. https://www.avg.com

[lefty665]

Hi Volker, Moving the mouse worked, thanks again for the heads up. Curious note. Not only does the mouse have to move off the server data list, but it also has to move entirely off the connect object so it loses focus before the table will sort. Then the sorted list appears quickly (nice optimization you've done there!) but the mouse has to move back onto the connect object and then onto the data list so it can regain focus before a server selection can be made. Objects, pains in the butt, but wouldn't want to live without them. Again, thanks for Jamulus and all you do. We had another good session this evening with pickers including two of us in Virginia, one in Maryland, one in Canada and another in Michigan on a server in the D.C. area. That's probably a 500-600 mile spread and delay was only slightly pesky. Good stuff. Regards, Cliff

…

On 10/22/2020 12:52 PM, Cliff Rowland wrote: Thanks, guess I've been too compulsive with the mouse, scrolling up and down the unsorted table cursing and looking for low ping times. ;-) On 10/15/2020 11:33 AM, Volker Fischer wrote: > > either not sorting by ping time > > If the mouse pointer is located over the table, the list is not > sorted. So you have to move the mouse pointer away from the list so > that it is getting sorted. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#255 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/APWAKF5EZJ6YNN4GJJSSNU3SK4I6JANCNFSM4NGY5CNA>. >

-- This email has been checked for viruses by AVG. https://www.avg.com

[corrados]

A long time ago I posted a question which is still open:

@softins It would be interesting to see how the new reduced server list performs if looking at the wireshark analysis.

I have not heard any positive/negative feedback since I implemented the reduced server list but the last Jamulus version is not out for a long time. I hope to read feedback from users soon which report that they now see a list were the previously had not seen a list.

Is there anybody who can comment on that?

[gene96817]

I have just now reviewed this topic and all the discussion. In my opinion:
1- It would be better to stay with UDP than to move to TCP.
2- It appears that some lost UDP packets would explain the problem.
A fix is for the table retrieval to include the table length and row numbers.
Then the client will know if rows are missing because of a lost UDP packet.
The client can then request just the missing rows. Then we don't care if there are links in the path that request short UDP packets (we really should not be sensitive to packet fragmentation).
3- I believe I am seeing bad table updates when there is congestion in the network path causing lost UDP packets. Perhaps changes in how pings are done at startup would help. Especially retrying pings if there is no response due to lost UDP packets.

[corrados]

Yes, Jamulus already tries over and over again to receive the list if it is not yet received. The problem for some people was that they never got any list. In your case, you get the list at some point in time.

[gene96817]

When retrieving the whole list fails, does Jamulus have a method to retrieve the list in parts? I was fishing for the distinction for (a) the retrieval only succeeds if the whole list arrives vs (b) if one of the UDP packets is lost, just the lost packet (or lost part of the table) is retried. When the probability of losing UDP packets is significant, then (b) has a better success rate. (b) is also much more tolerant of fragmentation (i.e. having lots of small packets).

[softins]

When retrieving the whole list fails, does Jamulus have a method to retrieve the list in parts? I was fishing for the distinction for (a) the retrieval only succeeds if the whole list arrives vs (b) if one of the UDP packets is lost, just the lost packet (or lost part of the table) is retried. When the probability of losing UDP packets is significant, then (b) has a better success rate. (b) is also much more tolerant of fragmentation (i.e. having lots of small packets).

Sorry, just got back into this discussion. Just a few comments on the above:

• The list of servers is only ever sent as a single UDP message, which may be as large as 6000-9000 bytes from a full central server. With such large packets, the IP message containing it must be fragmented by the networking layer, into IP fragments that are less than 1500 bytes in size, at a minimum, and possibly a bit smaller too. These are not individual UDP packets; they are component parts of a single UDP packet. If any of them is lost, the whole UDP message is undeliverable. This is why an affected client cannot display any of the server list. It cannot display just parts, it is all or nothing.
• When the probability of losing UDP packets is significant, the display of servers is the least of your worries. Under such network conditions, the audio quality will likely be too bad for playing anyway.
• The problem with IP fragmentation is not one of network quality, but rather of router configuration. A small minority of Jamulus users have routers which refuse to pass fragmented IP packets, or have an option to control this, which the user has not set correctly (probably due to not knowing it was there).

@corrados I haven't done any detailed Wireshark analysis of the reduced server list, because I have no problem with receiving and processing fragmented IP packets. I think I did look at it once when first implemented, and noted that it resulted in fewer fragments, but still did require fragmentation.

[gene96817]

@softins Thanks... I forgot some of the details about the router configuration.

[corrados]

I think I did look at it once when first implemented, and noted that it resulted in fewer fragments, but still did require fragmentation.

Thanks for the feedback. Yes, we still get fragmented packets but the hope is that we get fewer fragments and therefore fewer problems. Looking at the Jamulus discussion forums and Facebook groups, it seems that the complains about empty lists have not shown up recently (at least what I wrote) which is a good sign.

[lefty665]

No list was my issue. While I had no router settings that would ameliorate the problem, rebooting the router would. Thankfully that is rarely needed since your updates. Thanks again, Cliff

…

On 1/4/2021 12:01 PM, Volker Fischer wrote: Yes, Jamulus already tries over and over again to receive the list if it is not yet received. The problem for some people was that they never got any list. In your case, you get the list at some point in time. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#255 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APWAKFYH2QKCZIFUS6EIJPDSYHX6NANCNFSM4NGY5CNA>.

-- This email has been checked for viruses by AVG. https://www.avg.com

[gilgongo]

Hi All - I'm moving this to a discussion now until such time as we can firm up some actionable tickets on it.

[dtinth]

It seems that a lot of systems are blocking fragmented UDP packets and I have no control over them:

• VPNs
  • Cloudflare WARP
• Cloud providers
  • Microsoft Azure
  • Google Cloud
• Many apartment routers outside of our control

Would also 👍 to the idea of TCP-based directory server.

(Context: I was trying to deploy a copy of jamulus-php to some serverless providers. It turns out that the reply packets are being blocked for Any Genre 1 and 2, possibly due to the reply being too large…? Finally end up finding a local VPS provider that works, but it’s not serverless.)

[softins]

The directory list of servers is the main situation that results in IP fragmentation, but is not the only one. For a large server supporting many tens of clients, the client list can be large enough to need fragmentation too. Back during lockdown, I did see reports from people saying their client list went blank once more than a certain number of people had joined the server.

I'm reminded of the DNS protocol, where, if I remember correctly, a DNS server responding to a UDP request can indicate to the client that the response is too large and that the client should retry the request using TCP (I haven't studied the details). Maybe we could implement something similar in Jamulus?

Another idea I have wondered about but never tried is using zlib to compress server lists and client lists. There must be a lot of repeated compressible data, which could result in a much smaller payload. But it would probably need new message types to be defined.

The TCP fallback idea would probably work best for backward compatibility.

[softins]

Actually, what I think I would do is have the server listen for TCP on the same port as for UDP, and respond via the TCP connection to a request received on it.

It would handle requests from older clients the same as at present.

Then a newer client wanting a server list or client list could try the TCP connection first. And older server would just refuse the connection and the client would then fall back to the existing UDP method.

[softins]

Another idea I have wondered about but never tried is using zlib to compress server lists and client lists. There must be a lot of repeated compressible data, which could result in a much smaller payload. But it would probably need new message types to be defined.

I just did a test to see how compressible a typical long list of servers from a directory would be. I used the excellent jamulus-python from @passing, and added some test calls to compress the received payload with zlib at different compression levels:

tony@Tonys-MacBook-Pro jamulus-python % ./fetch_servers.py --directory anygenre1.jamulus.io
listening to port 22124
[0] ('62.113.204.104', 22124) <  #0 CLM_REQ_SERVER_LIST (9)
[1] ('62.113.204.104', 22124)  > #0 CLM_RED_SERVER_LIST (2008)
Compression = -1, payload size = 1587
Compression = 0, payload size = 2019
Compression = 1, payload size = 1592
Compression = 2, payload size = 1590
Compression = 3, payload size = 1590
Compression = 4, payload size = 1587
Compression = 5, payload size = 1587
Compression = 6, payload size = 1587
Compression = 7, payload size = 1587
Compression = 8, payload size = 1587
Compression = 9, payload size = 1587
[2] ('62.113.204.104', 22124)  > #0 CLM_SERVER_LIST (3704)
Compression = -1, payload size = 2517
Compression = 0, payload size = 3715
Compression = 1, payload size = 2572
Compression = 2, payload size = 2561
Compression = 3, payload size = 2549
Compression = 4, payload size = 2519
Compression = 5, payload size = 2517
Compression = 6, payload size = 2517
Compression = 7, payload size = 2517
Compression = 8, payload size = 2517
Compression = 9, payload size = 2517
add/update 98 servers

It shows that the directory lists are only moderately compressible, and would still result in UDP fragmentation.

So I have discounted that as an avenue to explore, as it would not gain us any advantage.