Sure you can argue that the tradeoffs they selected were the wrong tradeoffs and that they may have been biased by their narrow domain specific goals (reducing latency to render an average web page). Perhaps they thought that people with really horrible packet loss will just fallback to http or use amp or whatever.
But I cannot believe that the mistakes were caused by them just not being aware of how networks work or, worse, failing to talk with somebody that did
You can choose to start out with skepticism or cynicism if you like but there’s no defect in people who don’t.
Its nothing about faith here. I was looking at the evidence. The problem was that a lot of people I was working with had faith that SPDY was something that it wasn't.
They either interpreted what I was saying as either an excuse to not change infra (the change was minimal), "old think" because I was not a hot young fullstack node engineer, or that they thought I was trying to be more clever than google.
What seems more likely?
The main problem you're talking of is head of line blocking due to packet loss. But packet loss as a congestion signal is nowhere near as common as people think, and that was already the case during the original SPDY design window. End-user networks had mostly been set up to buffer rather than drop. (I know this for an absolute fact, because I spent that same time window building and deploying TCP optimization middleboxes to mobile networks. And we had to pivot hard on the technical direction due to how rare packet loss turned out to be on the typical mobile network). The real problem with networks of the time was high and variable latency, which was a major problem for H1 (due to no pipelining) even with the typical use of six concurrent connections to the same site.
Second, what you're missing is that even a marginal improvement would have been worth massive amounts of money to the company doing the protocol design and implementation. (Google knows exactly how much revenue every millisecond of extra search latency costs). So your "marginally better" cut isn't anywhere near as incisive as you think. It also cuts the other way: if SPDY really had been making those metrics worse like one would expect from your initial claims about H2 performing worse than H1, it would not have launched. It would not have mattered one bit whether the team designing the protocol wanted it deployed for selfish reasons, they would not have gotten launch approval for something costing tens or hundreds of millions in lost revenue due to worse service.
Third, you're only concerned with the downside of H2. In particular HPACK compression of requests was a really big deal given the asymmetrically slow uplink connections of the time, and fundamentally depends on multiplexing all the requests over the same connection. So then it's a tradeoff: when deciding whether to multiplex all the traffic over a single connection, is the higher level of vulnerability to packet loss (in terms of head of line blocking, impact on the TCP congestion window) worth the benefits (HPACK, only needing to do one TLS handshake)?
It didn't need to be the Holy Grail to be worth creating. It just needed to be better than H1 was, or better than what H1 could be evolved to with the same amount of effort. And likewise, it's totally possible for H2 to be better than H1 while also H3 is better than H2.
You appear to be confused by the idea that somebody would ever create something other than the absolute best possible thing. Why create H2 first, rather than just jump straight to H3?
One obvious reason is that H2 is far less complex than H3 thanks to piggybacking on TCP as the transport mechanism. The downside is that you then have to deal with all the problems of TCP as well. At the time, there would have been a lot of optimism about eliminating those problems by evolving TCP. An extension for connection migration, an extension for 0-RTT connection setup, an extension for useful out of order data delivery, etc.
It was only a bit later that it became clear just how ossified TCP was. Up until then, one could have told a story about how Microsoft controlled the primary client operating system, and were not really motivated to implement the RFCs in a timely manner, and that's why the feature rollouts were taking a decade. In the 2010s, it became clear that evolution was impossible even when all the coordination and motivation was there. See TCP Fast Open for a practical example.
So around 2015-ish you see big tech companies switch to UDP-based protocols just so that they can actually do protocol evolution.
The other plausible reason is that it's going to be far easier to get the initial work for making H2 funded, since the scope is more limited. And once you show the real-world gains (which, again, would have been there since H2 is better than H1), you then have the credibility to get funding for a more ambitious project.
I agree that H2 is defacto better than H1, and easier to implement when compared to H3. However, I'll call out the 2 biggest time sinks of the RFC: stream prioritisation and server push. Both of which had narrow application, and incomplete/inefficient specification. H3 seems to have ditched both. My question is, how did this ever end up in the final RFC? As both seem like the kind of thing that could have been easily disproved in SPDY deployments, as well as just asking people doing HTTP for a living.
It has almost _always_ been tied to OS, but moreover the OS of every node in between you and the webpage. That was the most frustrating thing, there were and are solutions for making TCP more latency resistant but also get better throughput and deal with "buffer bloat" which was a big thing at the time.
I was working in transcontental bulk transfer which used to mean that things like aspera/fasp was the defacto standard for fast/latency/loss resistant transport. So I had seen this first hand. I suspect it was probably why I was dismissed, because I wasn't a dyed in the wool webdev.
Surely packet loss due to poor signal quality is rather common over mobile networks and that packet loss still affects TCP's congestion window.
Admittedly anecdotal, but I just connected to a 5G network with low signal strength and it certainly seems to be the case.
The base stations and terminals are constantly monitoring the signal quality and adjusting the error correction rate. A bad signal will mean that there's more error correction overhead, and that's why the connection is slower overall.
Second,the radio protocol doesn't treat data transmissions as a one-and-done deal. The protocols are built on the assumption of a high rate of unrecoverable transmission errors. Those rates would be way too high for TCP to be able to function, so retransmissions are instead baked in at the physical protocol level. The cellular base station will basically buffer the data until it's been reliably delivered or until the client moves to a different cell.
And crucially, not only is the physical protocol reliable but it's also in-sequences. A packet that wasn't received successfully shows up just as one (radio) roundtrip latency blip during which no data at all shows up at the client, not as packet loss or reordering that would be visible to the TCP stack on either side.
Other error cases you'd get are:
- Massive amounts of queueing, with the queues being per-user rather than per-base station (we measured up to a minute of queueing in testing). The queue times would translate directly to latency, completely dominating any other component.
- A catastrophic loss of all in-flight packets at once, which we believed was generally caused by handover from one cell to another.
If you don't understand something, it'd be more productive to ask questions than just make shit up like that.
In principle you could observe it from some kind of OS level counters that aren't ACLd, but in practice the counters are not reliable enough for that.
Now, the things like "calls dropping" or "connections getting reset" that you're calling out have nothing to do with packet loss. It's pretty obvious that you're not very technical and think that all error conditions are just the same thing and you can just mix them together. But what comes out is just technobabble.
I'm saying that from the point of view of either endpoint, there is very little packet loss in mobile networks, because of error correction and retransmissions being handled at the physical layer. This is the third time I've written it. Both previous times you've not answered that, and instead made up a strawman about HTTP/2 and magic. Why do you keep doing that?
Do you not believe that cellular radio protocols do error correction? Or that they do retransmissions at that level, rather than just try transmitting each packet once and then give up?
Two points:
1. It's not commonly realised the TCP is terrible on lossy networks, where terrible means gets less than 10% of the potential throughput. It only becomes apparent when you try to use TCP over a lossy network of course, and most real networks we use aren't lossy. Engineers who try to use TCP over lossy networks end up replacing it with something else. FWIW, the problem is TCP uses packet loss as a congestion signal. It handles congestion pretty well by backing off. But packet loss can also mean the packet was actually lost. The right responses in that case are to reduce the packet size and/or increase error correction, but _not_ decrease your transmission rate. Thus two responses to the same signal conflict.
2. Because of that, the layer two networks the internet uses have evolved to have really low error rates, which is why most people don't experience TCP's problems in that area. As it happens just about any sort of wireless has really high error rates, so they have to mask it. And they do, by having lots of ECC and doing their own ACK/NAKs. This might create lots of fluctuations in available bandwidth - but that is what TCP is good at handling.
By the by, another reason we have come to depend on really low error rates on layer 2. That's because TCP's error detection is poor. It lets roughly one bad packet through in every 10,000. (Adler32 is very poor on small packets.) You can send 100,000 packets a second at 1Gb/sec, so you need to keep the underlying error rate very low to ensure the backup you are sending to Backblaze isn't mysteriously corrupted a few times a year. <rant>IMO, we should have switched to 64 bit CRC's decades ago.</rant>
> -these highly experienced guys actually knew what they were doing
> What seems more likely?
Well, when you put it that way... the former. By a large margin.
You're making quite clear you are the type of person who is extremely quick to accuse everyone and anyone of being incompetent in the absence of evidence or in spite of evidence.
You do not need to Google too hard to find tons of open-source benchmarks of real world servers showing off performance gains from switching to HTTP/2 and HTTP/3.
But here you are, claiming everyone is incompetent and that their work was bad. In spite of all the evidence.
It's clear that you have nothing relevant to say about the topic and no evidence to even suggest your beliefs have a leg to stand on.
These junior developers also have a tendency to make improvements to the system by implementing brand-new features instead of improving old ones. Look at recent Microsoft releases: we don't fix old features, but accrete new ones. New features help much more at review time than improvements to old ones.
(That's literally the explanation for PowerShell. Many of us wanted to improve cmd.exe, but couldn't.)
[0] https://blog.zorinaq.com/i-contribute-to-the-windows-kernel-...
(I'd heard the story in person from Snover years before that, so it definitely wasn't just sour grapes from somebody about to leave the company.)
I can totally believe that MS had a culture where making improvements to cmd.exe was impossible to justify, but that doesn't actually mean that a greenfield project in the same space would be considered promo-worthy either. It seems more likely that everything in that space was viewed as career suicide and you needed to be working on Internet stuff, just like at today's bigco anything except AI will have the same problem.
Isn't Windows Console the (old) terminal emulator, with cmd.exe being just the command-line interpreter.
Before that, csrss.exe handled terminal emulation (which is why console windows weren't themable[1]).
[1]: https://devblogs.microsoft.com/oldnewthing/20071231-00/?p=23...
How do you explain that some HTTP/2 server implementations handle an order of magnitude more connections than their HTTP/1.1 implementations? Is this something you downplay as accidental benefits of doing something to propel their career?
You should talk less about conspiratorial accusations you are making up on the spot and more about hard data.
As someone who was part of the rollout of HTTP2 for a $large_website, I can confirm that "this will harm mobile performance" was outright and flatly rejected. This included people who were our reps on the w3c. I just had to sit there and wait for the real world metrics to come in
"multiplexing will remove bottlenecks!"
"benchmarks prove that its faster!"
"you just don't understand how TCP works"
"the people at google are very smart, what do you know?"
"server push will reduce latency"
etc etc etc.
We even had the graphs of page size over time (going ever up) and average usable bandwidth (not keeping up, especially on mobile) None of that mattered until the rollout had a real world effect on our performance.
This sounds like a bullshit conspiratorial excuse. If you have real world data and you aren't afraid of having peers looking through it, nothing prevents you from presenting it to peers.
So where is that data?
Instead, you just have vague unsupported unbelievable claims made by random people in the internet, as if that's any way to decide over policy, and any faint doubt raised over that claim is faced with conspiratorial remarks complemented by statements on how everyone around OP is incompetent except him.
I will go as far as to claim OP's assertion is unbelievable, to the point of sounding like bullshit. It's entirely unbelievable that people designing protocols for a multinational corporation whose bread and butter is stuff done over TCP connections were oblivious to how TCP works, and the most incompetent of them would bother to design the first major revision of HTTP. Unbelievable.
But hey, some random guy online said something, so it must be true!
If that article really exists,if you read it, if you paid attention to it,and if you really felt it was relevant, you'd already have posted the link.
But no. You're just throwing distractions on how others can Google for stuff.
Enough.
If you don't want to be banned, you're welcome to email [email protected] and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.
If you don't want to be banned, you're welcome to email [email protected] and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.
https://bugzilla.mozilla.org/show_bug.cgi?id=779413
In their performance tests vs HTTP 1.1 the team simulated loading many top websites, but presumably by accident used a single TCP connection for SPDY across the entire test suite (this was visible in their screenshots of Chrome's network panel, no connection time for SPDY).
They also never tested SPDY against pipelining - but Microsoft did and found pipelining performed the same. SPDY's benefit was merely a cleaner, less messy equivalent of pipelining.
So I think it's fair to say these developers were not the best Google had to offer.
I think it's good that things have evolved rather than being stuck behind naysaying. Sure, there were pitfalls/hurdles that could have been avoided, but it's not clear that maintaining perfection every step of the way would have got us to QUIC.
The result of this is that we get constant upgrades and "improvements", but overall reliability, UI usability, latency and performance in general are actually worsening every year.
is this just a rose-tinted gut feel, or do you have actual data?
UI usability: This is a discoverable, good UI: https://jbss.de/bilder/scr118d.gif . You can instantly tell what is a button, where you can enter text and what is inert. You can even read out all the hotkeys from the screenshot. Now, without hovering over it with your mouse, check the header of this post (the grey line) and tell me which strings will take you to a place (= are a link), which do an action (= are a button) or are inert. They all look the same and you need to investigate it first with the cursor. And even then you can't tell the buttons ("flag", "vouch") apart from the links ("parent", "context").
Latency and performance: I played RTS games for 15 years. Despite getting old and rustly, i easily hit 120 APM during office work. I can tell if an application can handle it. Things like the Windows Start menu were fast enough for me until like Vista, but now they aren't - i need to wait a bit or i click before it is popped up.
The CPU/GPU required to render an interface at 60hz has gone up inline with moore's law.
Last I checked my old commodore PET wasn’t rendering ray-traced 4K UHD graphics at 60hz…
60hz is pretty easy if all you are doing is a simple character buffer.
Nor is my phone, yet newer apps are now struggling to render basic GUIs at 60hz.
While it's true that HTTP/2 can be worse than HTTP/1.1, I don't think it usually is; it's pretty easy to demonstrate just how much better HTTP/2 is over a typical Internet connection. SPDY and HTTP/2 were clearly better and rarely worse, whereas HTTP/3 is almost never worse (maybe when interplay with TCP rate control is poor?) On very unreliable and very high latency connections it can definitely go the other way, but statistically my experience is that a large majority of cases see an improvement on plain-old HTTP/2.
That said, for all of the complexity HTTP/2 adds, it is kind of a nice protocol. I like that all of the "special" parts of the HTTP/1.1 request and response were just turned into header fields. HPACK is a minor pain in the ass, but it is pretty efficient. You get multiple concurrent bidirectional streams per connection and they can each send headers/trailers. There's even the MASQUE protocol, which enables unreliable datagrams with HTTP/3. Put together this makes HTTP/2 and HTTP/3 amazingly versatile protocols that you can really use for all kinds of shit, which makes sense given the legacy of HTTP/1.1.
There are some pitfalls even still. For example, all of this added complexity has made life a bit harder for load balancing and middleboxes. TCP level load balancing or round robin is basically defeated by using HTTP/2 multiplexing, without the client being explicitly cautious of this.
It’s also a specious argument anyway. The six connection limit isn’t purely artificial, opening and tracking TCP connection state is expensive, and happens entirely in the kernel. There’s a very real cap on how many TCP connections a machine can serve before the kernel starts barfing, and that cap is substantially lower than the number of multiplexed streams you can push over a single TCP connection.
You’re also completely ignoring TCP slow start process, which you can bet your bottom dollar will prevent six TCP streams beating six multiplexed streams over a single TCP stream when measuring latency from first connection.
Oh and while doing that, also feel free to respond to the rest of comment that outlines why opening an unbounded number of TCP connections to a server might be a bad idea.
Besides that, more than half my point is that I like HTTP/2 and HTTP/3 for the featureset, and you can't get that by increasing the max connection limit for HTTP/1.1.
When the limit is reached, all clients would be broken
Looks like a good deal indeed, why do not we do that
All browsers cap the number of connections which are opened to a single domain (I think on IE this was 4, and has increased to 10, but it's not a large number).
[1] when there is high latency, or some packet loss, and the requests are batched evenly over all connections.
1. This is text retrieval. The www is not a handful of browsers controlled by companies that seek to profit from advertising. It was and still is a facility that provides for (hyper)text retrieval.
HTTP/2 and HTTP/3 come from an advertising commpany and a CDN that expect ads hosted on different domains in every page. That is what _they_ want. Is that want _www users_ want. We do not know because www users were never asked. We do know that www users do not like ads. When given the choice, they say, "No."
The advertising company keeps repeating this HOL blocking as a problem of HTTP/1.1, and now it gets parroted everywhere, and few even know what it means. HOL blocking is not a problem if the www user is not requesting ads and tracking from different domains. How many www users actually want to request ads and tracking and particpate in telemetry. The so-called "tech" company might try to argue that all of them do, or more commonly that, "They do not care." Meanwhile its own employees call ad blocking a "right of passage" (direct quote from a reply I got on HN).
The truth is that when evaluating these new HTTP protocols, it matters what the www user is trying to do. Many users are simply trying to retrieve information as text. But the advertising company believes that www users only want to do what is in the interest of the advertising company: let the advertising company's browser automatically send requests for ads, tracking and telemetry purposes. (Except for employees of so-called "tech" companies profiting from the sale of online ad services. They exempted and are free to block the ads and tracking.)
Agreed - i've always found them to be living in a world of their own, which doesn't match the realities of actual networking out there.
In fact, the very same team after touting QUIC over UDP as the revolutionary protocol is now complaining that the real world realities aren't matching their expectations and so are now proposing to do QUIC over TCP. Here's that proposal https://mailarchive.ietf.org/arch/msg/quic/N82WBOa_RJIb4cPQw...
Thats not really head of line blocking, because in HTTP1.1 you'd just open up another connection. The issue with HTTP1.1 is that opening up lots of connections can introduce lots of latency, especially if you are doing Encryption.
HTTP1.1 performs much much better over high latency or lossy links than http2
> With HTTP/2, this problem is solved with streams, each stream corresponds to a message. Many streams can be interleaved in a single TCP packet. If a stream can't emit its data for some reason, other streams can take its place in the TCP packet.
This is where HTTP2 failed. It shoved everything into one single TCP connection which works fine on LAN and LAN like networks, and sucks balls in the real world. This is Top of line blocking and was entirely predictable had the HTTP2 team bothered to talk to anyone who did networking.
Its part of the reason why I was greatly suspicious of QUIC, because it appeared like it was designed by the same people that thought http2 was a good idea.
However QUIC seems to be actually reasonable. I've yet to fully test it in real world scenarios, but it does offer promise for highspeed latency resistant data streaming. One day I'll re-write my TCP multiplexor to compare the performance.