Re:Etcd, or, why modern software makes me sad
by derefr on Wednesday July 15, 2020 @11:43AM
(#23846858)
It took me a bit to understand what you were saying here:
> I'm talking about not putting control signals in the same TCP connection that you are shoving tonnes of data down. That's never going to scale past 70-100ms.
...but I think I got it now. I was arguing with the wrong thing here.
Yes, there's a reason that FTP and RTP do what they do. (Though in FTP's case it had more to do with TCP-port parity gender; multiple data sockets per control socket was just a convenient outgrowth of the existing architecture.)
But crucially, HTTP/{2,3} are about accelerating existing traffic; and existing HTTP "apps" (the HTML + JS + CSS that contain an implicit dependent-resource digraph that gets turned into flow-requests by the browser) have no indication in them for what's going to be a "control-plane" request vs. a "data-plane" request. HTTP itself is an oblivious carrier re: these use-cases. So HTTP/{2,3} must continue that trend, and be oblivious carriers as well. To do otherwise would force a rewrite of all HTTP "apps" to add such indications.
However, in HTTP/1.1, you got the benefit of this purpose-isolated signalling "for free" anyway, because you'd probably be making your data requests from a CDN origin, while making control requests to an API-subdomain origin. Those are separate origins, so they got separate (keepalive, pipelined) TCP sockets.
In HTTP/{2,3}, you can choose to shove everything into one socket by putting both your control-plane and your data-plane into the same origin.
But, crucially, you don't have to. Nobody's stopping your HTTP "app" from just having separate control and data origins; and so ending up with two HTTP/{2,3} sockets open, one for control signals, the other for BLOB data.
And, of course, existing sites previously engineered for HTTP/1.1 won't shove everything into a single socket, since they've already separated the control origin from the data origin. (They've likely got several data origins, so they'll likely want to re-engineer with upstream proxying to compact those down into one data origin; but all they have to do while doing this is to choose not to also compact the control origin in there as well. The choice to not change anything is, in this case, the right choice!)
And also keep in mind that most websites aren't, like, photographer's portfolios or direct video hosts. There are few websites directly embedding same-origin BLOBs. There are many websites embedding "Binary Not-so-large OBjects", like blogs that embed same-origin 500px-wide medium-quality optimized-JPEG images. And those websites don't have such a large disparity between control signalling and data signalling that they have any need to separate them—those images aren't much larger than the average compressed-JSON XHR response!
And because of that, the "default" advice for HTTP/{2,3} transitioning (that you should follow if you're 99% of websites) can just be "put everything on one origin"; whereas "split control signals to a different origin, thereby forcing it to a separate socket with separate TCP window scaling parameters" can just be one of those arcane PageSpeed things you learn from an infrastructure consultant, a bit like choosing what type of select(2)-alike kernel ABI your webserver should use to receive packets.)
> I'm talking about not putting control signals in the same TCP connection that you are shoving tonnes of data down. That's never going to scale past 70-100ms.
...but I think I got it now. I was arguing with the wrong thing here.
Yes, there's a reason that FTP and RTP do what they do. (Though in FTP's case it had more to do with TCP-port parity gender; multiple data sockets per control socket was just a convenient outgrowth of the existing architecture.)
But crucially, HTTP/{2,3} are about accelerating existing traffic; and existing HTTP "apps" (the HTML + JS + CSS that contain an implicit dependent-resource digraph that gets turned into flow-requests by the browser) have no indication in them for what's going to be a "control-plane" request vs. a "data-plane" request. HTTP itself is an oblivious carrier re: these use-cases. So HTTP/{2,3} must continue that trend, and be oblivious carriers as well. To do otherwise would force a rewrite of all HTTP "apps" to add such indications.
However, in HTTP/1.1, you got the benefit of this purpose-isolated signalling "for free" anyway, because you'd probably be making your data requests from a CDN origin, while making control requests to an API-subdomain origin. Those are separate origins, so they got separate (keepalive, pipelined) TCP sockets.
In HTTP/{2,3}, you can choose to shove everything into one socket by putting both your control-plane and your data-plane into the same origin.
But, crucially, you don't have to. Nobody's stopping your HTTP "app" from just having separate control and data origins; and so ending up with two HTTP/{2,3} sockets open, one for control signals, the other for BLOB data.
And, of course, existing sites previously engineered for HTTP/1.1 won't shove everything into a single socket, since they've already separated the control origin from the data origin. (They've likely got several data origins, so they'll likely want to re-engineer with upstream proxying to compact those down into one data origin; but all they have to do while doing this is to choose not to also compact the control origin in there as well. The choice to not change anything is, in this case, the right choice!)
And also keep in mind that most websites aren't, like, photographer's portfolios or direct video hosts. There are few websites directly embedding same-origin BLOBs. There are many websites embedding "Binary Not-so-large OBjects", like blogs that embed same-origin 500px-wide medium-quality optimized-JPEG images. And those websites don't have such a large disparity between control signalling and data signalling that they have any need to separate them—those images aren't much larger than the average compressed-JSON XHR response!
And because of that, the "default" advice for HTTP/{2,3} transitioning (that you should follow if you're 99% of websites) can just be "put everything on one origin"; whereas "split control signals to a different origin, thereby forcing it to a separate socket with separate TCP window scaling parameters" can just be one of those arcane PageSpeed things you learn from an infrastructure consultant, a bit like choosing what type of select(2)-alike kernel ABI your webserver should use to receive packets.)