Field report -- OpenWRT 23.05 virtual router on Intel N100 running Proxmox PVE 8.0 #274
Labels
Comments
|
Network configuration Waveform.com bufferbloat test results at close range -- about 4m away OOKla SpeedTest results for reference: 2. WIth SQM, download limit 650 Mbps, upload limit 380 Mbps -- A and A+ https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm Queueing discipline: cake https://www.waveform.com/tools/bufferbloat?test-id=ead5bb1d-b6c4-47c6-805b-c4ca2a083661 3. WIth SQM, download limit 650 Mbps, upload limit 200 Mbps -- A and A+ |
|
Just wondering what is the better way to carry out the testing. Thanks. I feel waveform.com test data may be on the low side so I am not so sure if the result is meaningful or not. I may want to try using Reference: |
|
Waveform.com bufferbloat test results with a thick wall in between laptop and the router
|
|
Using crusader to carry out bufferbloat test results at close range -- about 4m away, test over the internet. For my setup I can only test download and upload one at a time, not at the same time, as the two networks share the same GPON ONT connection.
|
|
@richb-hanover and @lynxthecat Just wondering how do I use crusader (or other tools) to demonstrate the effectiveness of I the above post, I can see the upload speed is ramping up. Is that the idea of |
|
@moeller0 is very good on the analysis side of things. Perhaps he has some thoughts. Please bear in mind that as detailed in its description, cake-autorate is designed for variable bandwidth connections in which the bandwidth fluctuates with time and hence the cake bandwidth must be continually updated. For fixed bandwidth connections, the best that cake-autorate can achieve is to not unduly hurt performance set against simply setting the correct fixed bandwidth. Perhaps this analogy helps - you have a drainpipe and you want to pour water into at the highest flow rate possible. If you know the capacity in advance, you just pour in the water at the correct maximum flow rate. If you do not know the capacity, you test it by pouring in water at an ever increasing flow rate until you get some overflow, and then, in response to the overflow, you reduce the flow rate a little. Thus testing for the correct flow rate involves some degree of: 1) lost flow rate; and 2) overflow. If you already knew the capacity in advance, testing for the flow rate leads to unnecessary loss in flow rate and overflow. In cake-autorate, the flow rate is bandwidth and the overflow is increased latency. And just as in our water analogy in which unnecessarily testing for the capacity results in unnecessary loss in flow rate and overflow, unnecessarily testing connection capacity results in unnecessary bandwidth loss and latency increase. |
Thanks for the explanation and analogy. Yes, the reason to look at cake-autorate is because wireless connection at home is of variable bandwidth. The above test results are all using wireless. For example, for my 1Gbps Fibre internet connection, wired download/upload speed will be close to 1Gbps (about 947Mbps). Close range wireless with the Asus RT-AX86U will be like about 500Mbps to 700Mbps, one thick wall away it will further reduce to about 300Mbps to 450Mbps (but still with good wireless signal). The idea of cake-autorate is good. I do not want to limit the bandwidth to 200Mbps to achieve good bufferbloat result, rather I'd like to have an adaptive bandwith limit based on the connection, no matter it is at close range or far range (variable speed), within a reasonable limit. Of course there will be a limit -- when the wireless signal is really weak, then I do not believe any SQM measure can do the magic. |
|
From the curve, I think I have some ideas to further carry out the test using crusader.
|
|
Changing the Configuration Test results at close range with
Download
Upload |
|
@moeller0 @Zoxc Network configuration: dual home network, but share the same GPON ONT Singtel GPON ONT -- TL-SG105E smart switch -- OpenWRT virtual router 1 WAN (2.5G port) OpenWRT virtual router 1 LAN (2.5G port) --Asus RT-AX82U AP --wireless -- Acer Windows 11 laptop using Intel AX201WiFi 6 adapter, as crusader client |
There is a lot to digest in your posts and I've been skimping, but I thought latency variation associated with WiFi is already addressed elsewhere (in WiFi drivers(?) - @moeller0 can hopefully confirm). Otherwise don't forget that for cake-autorate to work in that sense it would have to be running at the right location, i.e. in this case on the client device. Because if you run cake-autorate on your upstream router, the upstream ICMP responses will not reflect any latency increase associated with downstream WiFi latency. That is, for cake-autorate to work, the ICMP responses must reflect the delay you are seeking to address. I think you may already be doing that, but I just want to check. And probably you might want to play with the delay thresholds. |
So for local WiFi the active measurement approach that cake-autorate uses is not well suited*, it is much better to switch the APs over to models that support AQL/airtime fairness... that still leaves the stations only partially fixed *) each delay measurement, in spite of its small size will each up an airtime transmit slot, and airtime properties that determine the MCS likely can change on shorter timescales than individual latency measurements, so cake-autorate might be at the same tome too intrusive and yet too slow...
Maybe collect a cake-autorate logfile, which will allow to nicely see most relevant data simultaneously? |
Ah very good point. In this case, Overall Latency = WiFi related (WiFi client device to wireless AP) + upstream router related (OpenWRT in this case) + Internet (ISP related)
Yes, to play with the delay thresholds is in the plan as well. |
I see. Thanks for the answer. I was hoping that To use an AP which supports AQL/airtime fairness was the plan. However, it is not easy to find a wireless router with OpenWRT support yet can do SQM at 1Gbps. Reference from @richb-hanover's discussion thread in OpenWRT -- he was only looking at SQM > 300Mbps. I live in Singapore where 1Gbps Fibre becomes the lowest offering now that most of the ISPs have removed 500Mbps plan. Lower cost 10Gbps plan have been announced and 10Gbps plan is expected to become the main stream in 2028 (expected to hit 50% household in the end of 2028) because of the government's push for 10G internet (along with mobile 5G). In this case, I tend to think x86_64 mini PC as the main router running OpenWRT/pfSense + dump AP is the way to go.
I will do that. |
I am still interested to know your answer here. Thanks. Reference: looks like the WiFi driver is an important part of the story. But not so sure if there are other factors. |
Here is the log. Corresponding crusader test (09:16 am) using another wireless client (Windows 10 Dell laptop, Intel AC9560 WiFi 5 160MHz client). The other setup is the same as before.
OpenWRT SQM setting is as followed. |
|
Just as a comparison for another two tests.
|
|
@moeller0 and @lynxthecat Based on the above results (all using wireless), it seems to me that |
Using wired test (gigabit Ethernet) to see the side effect.
|
|
I live in Singapore where 1Gbps Fibre becomes the lowest offering now that most of the ISPs have removed 500Mbps plan. Well, nobody forces you to set the shaper close to the true speed ;) |
I agree that incorporating the correct technology into the WiFi stack is the right approach (at least for router initiated transmissions) and here Linux/OpenWrt offer this on a set of WiFi chips already:
I mean the full logfile, @lynxthecat what is the current way of exporting that again? I think that logging needs to be enabled in the configuration. |
|
@mcuee actually one of the earliest adopters of cake-autorate during its initial development successfully used it in exactly this scenario. I think he might have set the upstream router IP as a single reflector and so you could try that. To enable logging just enable the data and load log lines in the config and then you can use the auto-generated scripts in the /var/run/cake-autorate/primary/ folder to reset the log file, run tests, and export the log file as described in the analysis section of the README. |
|
Thanks. I need to learn how to analyze the data. Here is the configuration. Here is the crusader output (10 seconds output). Here is the log file. Analysis to be done later. |
I just installed Octave under Windows 11 but somehow it does not work. I will check under Linux later. |
|
Which octave version? I believe '.-' got deprecated in recent versions... |
Latest release version: 9.1.0. |
|
I think there are a few '.OPERATOR' instances, as a quick fix, just remove the leading dot from the operator... |
It seems to hang. |
|
I note the issue is close to the end (in the last DATA record) so relatively easy to ignore, yet, this should not happen... |
|
@moeller0 and @lynxthecat BTW, the document here may need a minor note about scp command, for macOS users. I just found it out on my Mac Mini M1 running latest macOS 14.4.1. Reference: need to use |
FYI I was using the following commands under OpenWRT. Before the test: After the test: Then I was using WinSCP to copy the log file to my Windows machine. Under Linux and macOS I will use the script. |
|
I created a PR with the required changes to robustify the parser. The parser will now try to skip incomplete lines but attempt to continue parsing beyond the problematic lines, hiw well this works is to be seen... |
|
Another thing, it is better to change the document as well to export png file and not pdf file. Exporting pdf file is not working under Windows, it is even worse under macOS. |
As far as I can tell that is the recommended work-flow. I have not looked into the actual shell code in ages, so I have no theory what exactly is happening, bt the phenotype looks like some writes do not finish and leave incomplete entries (also lacking the final line feed). I tried to make the parser hobble over those lines, but this is really just papering over the issue (yet, might be a good enough solution). |
Great idea, but I am not one of the maintainers of cake-autorate, so I think @lynxthecat or @rany2 should do this, if they agree. The issue is that with more recent octave/ghostscript combinations the pdf export of the timeline plot results in a blank file and the above error message... |
I think this is not an macosx issue, but will affect all users of recent enough OpenSSH... |
|
I've updated the instructions, thanks for the heads up and ping @moeller0. |
Please also update the following command to export png and not pdf file. Thanks. https://github.com/lynxthecat/cake-autorate/blob/master/ANALYSIS.md --> |
|
@mcuee Why? Does it rely on latex or something? |
|
@rany2 the issue is that recent octave/ghostscript combinations crash while trying to export the timecourse plot... the png exporter however does seem to work... |
|
Thanks, I updated the document now |
Ah yes, I see the issue. Fixing now. |
|
@lynxthecat In the old code, I had it acquire a lock for log_msg to avoid this situation; was the overhead too great that it had to be removed? |
|
Now it seems to be much more stable now -- all log files can be parsed. Run log for a OOkla Speedtest (wireless) Log file Generated pictures by Octave.
|
In this case it was the fixes tp the code that wrote the log, the robustification of the log file parser did not matter/trigger here at all. Which IMHO is how it should be, thanks for helping getting both sides improved. Looking at your speedtest, the Ookla servers did not manage to reach your shaper limit, but the overall mechanism seems to have worked well... similar for the Upload... |
Just wondering if I should lower which of the following parameters? I am reluctant to drop the max rate (that is the limit for wired connection). Shall I reduce the base rate? Full configuration file: |
|
I defer to @lynxthecat and other's that actually use cake-autorate in earnest... |
|
Perhaps try: For setting the base rates, it is a matter of personal preference. But I recommend thinking of the base rates as best set to safe rates where you will never or at least mostly not experience any latency increase on connection saturation at those rates. You don't want cake-autorate to keep setting the shaper rates back to a dangerous rate that might incur latency spikes on high load. Rather you want cake-autorate to carefully allow for bandwidth excursions from that area of safety. And once the load drops, cake-autorate should bring the shaper rates back down from such excursions to the safe rates. Hope this makes sense. |
|
Now there seems to be a problem to capture the download side. Strange. Configure file Runing OOkla SpeedTest from the wireless client. Log file generated. Analysis output
|
|
I guess posting the output of |
Sorry for the false alarm. I accidentally start another OpenWRT VM (older clone of the main OpenWRT VM) and the thing is not running correctly with the two VMs running in parallel.. |
Hmm, still the same results after I shut down the second VM. |
|
After reboot. OOkla SpeedTest: Log file: Output: |
|
Something is wrong with your native sqm/cake set-up, we only see a single cake instance in |
You are right. I do not know why it happened. I just changed the SQM settings from Luci GUI to restart the sqm/cake setup. Now it seems to be okay. OOkla SpeedTest. Log file generated: Analysis Output
|
I will need to carry out more testing and analysis, but it seems to help the situation at far range. With the following settings, it seems to me that I get better scores with waveform.com bufferbloat test. On the other hand, since waveform.com gets much lower speed in close range than OOkla SpeedTest, we may need to take the results with a pinch of salt. Close range: A+ now (used to be A without cake-autorate) One thick wall in between: B and C (used to be F without cake-autorate) Two walls in between (weak signal area): F and C (used to be worse F without cake-autorate) |
|
I will close this issue for now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
PVE 8.0 host: Intel N100 mini PC with quad Intel I226-V 2.5G adapter
OpenWRT virtual router: using two cores. no network passthrough, just use the Linux bridge. 4GB RAM, 4GB Storage.
cake-autorateConfiguration:Example run-log on the router.
OOkla SpeedTest run log on an Windows 11 laptop connected wireless to the Asus RT-AX82U AP which is connected to the OpenWRT virtual router LAN port.
CPU load from htop:
Download -- peak load: one CPU goes to near 100% where the other CPU is less than 20%.
Upload -- peak load: one CPU goes to about 40% whereas the other CPU is about 35%
waveform.com bufferbloat test results: Speed is on the low side so may not be that useful.
https://www.waveform.com/tools/bufferbloat?test-id=b47574e2-385b-4252-8dc3-e4b04a46c4e5
https://www.waveform.com/tools/bufferbloat?test-id=be87e694-b9ed-42fa-b0ed-db5185c372a7
The text was updated successfully, but these errors were encountered: