and I am curious if you have considered a fq_codel-like approach to message queuing? Sending a whole socketbuf kind of scares me.
'Course, I still kind of miss scan-lines, and had hoped we'd finally all have enough bandwidth to just blast raw frames over the network by now.
/me hides
There was a time, back in 1999 or so, when people were filled with revolutionary fire about the future of the internet and the merits of open source code. Recently, though, in light of so many abuses of binary blobs, like in the volkswagen scandal, and now Huawei, and shenanigans in the IETF, it feels as though we've lost our way.I wrote this song at the famous 1999 linuxworld conference, where...
I hope to have a document comparing it to docsis 3.1 pie at some point in the next few months, in the meantime, I hope more (especially ISPs in their default gear) give cake a try! It's open source, like everything else we do at bufferbloat.net and teklibre.
http://flent-newark.bufferbloat.net/~d/Airtime%20based%20que...
Ericson, at least, published a paper showing they recognized the problem: https://www.ericsson.com/en/ericsson-technology-review/archi...
and I do hope that shows up in something, however the chipsets on the handsets themselves also need rational buffer management.
Smaller buffers are in general better. However advanced AQM algorithms and fair queueing make for an even better network experience. Being one of the authors of fq_codel (RFC8290), it has generally been my hope to see that widely deployed. It is, actually - it's essentially the default qdisc in Linux now, and it is widely used in quite a few QoS (SQM) systems today. The hard part nowadays (since it's now in almost every home router) is convincing people (and ISPs) to do the right measurement and turn it on.
https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_d...
make-wifi-fast was this project where we took a deep dive into wifi, and cut 100x of the latency out across all rates and improved throughput on our tests by 2.5. That damn "fq bug" was the not the only, but certainly the worst bug we dealt with. Even harder was spending about 2 years, stumped, trying to come up with an efficient implementation of the fq_codel theory, in wifi. 5 years of work to get to this goal, with no progress til the end. No corp would have supported us...
This past year's project was sch_cake, which helps defeat bufferbloat on really slow connections. That will be in linux 4.19, writeup here: https://lwn.net/Articles/758353/
None of our projects ever attract funding, yet we do somehow manage to ship something great every other year.
After every "improvement", new problems emerge. We enabled ecn universally in the fq_codel for wifi code, in sch_cake, and partially in the sqm-scripts... then apple enabled ecn universally in their TCPs... last year... and I'm concerned now about how well that is working out. ECN was a "debugging aid" for us, mostly.
Our writeup of how it works is here: https://arxiv.org/pdf/1703.00064.pdf - lots of pretty pictures.
I recently had to shut down my lab for lack of funding.
1) If we had sufficient backpressure in the ISPs provided router network driver, with "BQL" to manage the ringbuffers and fq_codel to do fq + aqm, we're mostly done, at least on the uplink.
The hope has been, since most home routers run linux, that 5 years after RFC8920 entered mainline linux, it would appear in ISP gear. fq_codel is now the default on most linux distros but without the backpressure from bql or running below line rate it's not effective unless further configured or shaped.
2) I wish having your own shaper was not "a trend". bql and fq_codel are lightweight compared to shaping.
3) sch_cake solves a bunch of remaining problems: A) doing both flow and host based fairness at the same time, so your host frantically issuing http requests only gets 1/hosts the bandwidth. B) uses a deficit shaper that consistently runs at a rate slightly less than the (ubiquitous) token bucket shaper ISPs use. So it can share the same setpoint but control the queue.
https://arxiv.org/abs/1804.07617
4) we have a lot of wifi devices now doing fq_codel by default. Very happy with the results. https://arxiv.org/pdf/1703.00064.pdf
Perhaps on rfc970's 40th anniversary, we can view these problems as solved!
I was happy to wake up a few weeks back, and find RFC8290 published, and sch_cake being readied for mainline, and multiple commercial products finally shipping what we'd worked on all these years.
I am always happy to hear of a bloat free connection.
There's probably over 2b routers without bufferbloat fixes installed, and only if we work together to get them deployed will the internet as a whole get better, more capable of handling web, games, videoconferencing, and other applications that demand consistent low latency.
Thanks.
pfsense has fq_codel.
Anything (1000s of routers) from lede/openwrt has the most advanced bufferbloat-fighting stuff in it, followed by dd-wrt, tomato, etc. If you need high bandwidths the multi-core arms are the best. fq_codel is pre-configured on all links (ethernet/usb/fiber/wifi/whatever) but if you need shaping to the ISP provided rate you need to configure it. All the research that went into fixing bufferbloat queuing problems everywhere landed in openwrt and lede first. Most of the research that improved tcp everywhere came out of google.
Most gaming routers now sold commercially have some variant of fq_codel in them in their trade name ISP "qos" system.
Also fq_codel derived anti-bufferbloat work has landed in many commercial wifi routers on the wifi side (eero, google wifi, some ubnt products, meraki, many others). The paper behind all that was: https://arxiv.org/pdf/1703.00064.pdf - happily that work was "good enough" to enable by default, and boy, does it make a difference if wifi is your bottleneck.
The current premier dsl router with cake is evenroute. I think there are several new models from several manufacturers that are going to get it right, soon.
Not tracking FIOS (gpon fiber) closely at the moment. Yes, fiber networks have bufferbloat, but it's harder to hit, and generally smaller than on dsl and cable technologies. I configured cake on sonic fiber recently and got 60ms back. (going from 60ms latency under load to 3ms )
Regrettably shaper setup is finicky and requires a few minutes of testing with a site like dslreports or a tool like flent.org to get right. If more ISPs published their shapers' bitrate and burst rate settings, life would be easier here... but the hope has always been they'd just ship a router with this stuff on and remotely configured to be "right".
In some ways, what are you doing about bufferbloat is
We use diffserv for this, for apps willing to use it. Example: ssh sets the imm diffserv bit for interactive use. cake respects that (I've cited the relevant paper elsewhere, another place is https://www.bufferbloat.net/projects/codel/wiki/CakeTechnica... but after extensive testing we settled on 3, rather than four tiers of priority)
stuff derived from the sqm-scripts use the same method (using htb + fq_codel) but the problem has always been that diffserv is not respected end to end. However, within your network, you can make your intention known and have it work, if you have the bottleneck.
Also, we have always made the latency/bandwidth tradeoff explicit - if you want less latency, you must want less bandwidth. It's the only safe answer to apps gaming the diffserv markings.
1) BBR is currently not something I'd recommend at home.
2) The hope has always been that the core two bufferbloat-fighting algorithms (BQL, and fq_codel) would end up in the cable, fiber or dsl modem hardware, so that no shaping would be required, as there would be sufficient backpressure from the link itself to regulate the link intelligently. The cpu costs on this are nearly 0! BQL is 6 lines of new code in the device driver. fq_codel has been shipping in linux for 6 years. It's just a matter of turning it on...
But: lacking that support from the ISP-supplied gear, we shape with htb + fq_codel, (as you say), to ~90% of the link rate... with another box - or even in the same box if the device driver can't be fixed and is overbuffered. We are painfully aware of how much cpu shaping costs but modern cpus usually have enough oomph to handle it.
btw: We've come up with a new deficit shaper (sch_cake) that lets us get to ~100% of the isp bandwidth (so long as you get the wire framing exactly right), while providing vastly better queue management in the sqm system.
4) fq_codel is fair to flows, not devices. This works well in the general case, but has edge cases where abusive apps that open a lot of flows gain priority. Adding per host fq (while retaining per flow fq), even through nat, was the number 1 request from the users for sch_cake, and one of the main reasons why cake exists.
there's an awful lot of lit on FQ, what we do with fq_codel is to not only interleave packets better but apply congestion control signals at the right time so competing tcp flows don't overwhelm the link (with under 10ms of buffering (v seconds common on fifo ISP links)).
https://en.wikipedia.org/wiki/Fair_queuing
Of course, being perfectly fair to flows is sometimes undesirable, but making something strictly higher priority[1] is fraught with peril as you end up with a classification nightmare.
Having fq gives you the best shot at smaller flows completing sooner, and of big flows sharing better with each other.
Having vastly reduced buffering improves the responsiveness of competing TCP flows a lot, grabbing bandwidth whenever available, faster.
My take on folk that want "prioritization" is ask them to try some variant of sqm with just fq and codel and get back to us. being fair with well managed buffers works really well.[2]
[1] making something lower priority than best effort is actually a good idea. [2] but if you really want some flows or devices prioritized, see the sch_cake work mentioned on this thread. I still tend to think per host FQ is what many want rather than attempting to raise the priority of certain flows from certain services.
IF things like fq_codel were deployed on those we'd not see latencies climb that much at all, we'd see bandwidths decline to the actual capacity available - and only the biggest flows would be hit to do so.
FQ_codel is lightweight enough to fit directly in high speed hardware and it does indeed run on 40gigE plus devices on linux, ddpk, BSD, in software. But it takes a long time for new chipsets to adopt new algorithms even if they incorporate support for deeply desirable features like ECN.
That said... 10Gbit to the home.... ooohhhhh. it's really hard to bloat that!
QoE is a better, less overloaded, in the "qualitative, perceived" sort of description.
So (after endless debates) - we defined SQM as as a superset of classic as-defined-for-internet QoS and hoped for the best. (https://www.bufferbloat.net/projects/cerowrt/wiki/Smart_Queu... )
There's been plethora of other trade names for what we do with htb+fq_codel - streamboost, adaptive qos, etc.
I like that eero, edgerouter, and openwrt and derivatives also call what we do sqm. It simplifies the discussion, and the core scripts for linux generically are available as the sqm-scripts on github.
I also hope we see more RFPs specifying RFC8290.
“SQM” is shorthand for an integrated network system that performs better per-packet/per flow network scheduling, active queue length management (AQM), traffic shaping/rate limiting, and QoS (prioritization).
“Classic” QoS does prioritization only.
“Classic” AQM manages queue lengths only.
“Classic” packet scheduling does some form of fair queuing only.
“Classic” traffic shaping and policing sets hard limits on queue lengths and transfer rates
“Classic” rate limiting sets hard limits on network speeds.
It has become apparent that in order to ensure a good internet experience all of these techniques need to be combined and used as an integrated whole, and also represented as such to end-users."