QUIC's connection migration feature is really amazing for applications where you switch IPs, e.g. when your phone goes from mobile data to Wifi. Another nice thing is that you can cache sessions and resume them later, so connections can even survive reboots and server failures, or you can move them between different machines.

TCPLS seems like a good proposal but I won't hold my breath for it as the big players like Cloudflare, Google, Akamai, Fastly are already committed to QUIC. There's still significant churn in the proposals and some features like MASQUE are still being finalized, but overall QUIC works really well.

I'm not a huge fan of how spdy and then quic were kinda ramrodded into the HTTP standards process, but realistically I think QUIC is basically the future of connection transport wherever tcp or udp alone are failing at this point. I recently discovered the nQuick proposal to do noise-based connection initiation for peer-to-peer connections and I love it[1]. It's an incredibly versatile protocol.

It seems likely that a lot of the issues raised in the OP will be somewhat forced to be resolved due to the aforementioned ramrodding of QUIC into the HTTP3 path, but maybe in the long run the two ideas will co-evolve to the same solution.

[1] https://eprint.iacr.org/2019/028.pdf

> quic were kinda ramrodded into the HTTP standards process

QUIC took 4 years and 34 drafts to get standardized. The final IETF version is significantly different from Google initially came up with. So, no it most definitely was not ramrodded through.

I didn't say quic itself was -- if quic had been introduced completely independently of http then sure, that's fine. But it was introduced intermingled with making yet another drastic change to a backbone protocol of the internet.

Considering HTTP/1.1 stood for over 15 years before http/2 came along (and was a pretty modest improvement over http/1.0 compared to what we're talking about now, so I'm being generous by not saying over 20 years). http/2 had only been promoted to a proposed standard for 3 before http/3 was proposed. That, to me, is a very aggressive process. But again, it's the relationship to http that bothers me, not the standardization process for quic itself.

What's the point of having QUIC if there isn't a protocol that actually uses it? And what would be a better protocol than HTTP? HTTP/2 was a marginal improvement but HTTP/3 and QUIC significantly improve user experience. Especially if the users have jittery networks which is the case for the majority of the world. I don't understand why it would have been a better idea to sit on these improvements for a while before getting them into the hands of users.

I don't really see "is it beneficial in the long run" and "is it an aggressive standardization campaign by one interested party" as being the same question.

At any rate I also didn't say they shouldn't experiment with using it as an accelerated path for http. I didn't think they shouldn't with spdy either. But historically RFCs were built on practice, not the other way around.

If I were the Internet Dictator, I'd be a big fan of standardizing the semantics of http separate from the physical transport and then defining best practices for using alternative transports in the wild, and then eventually standardizing what comes out of that process.

To me it's an indictment of how all this shook out that we're so quickly looking to replace http/2. I think moving to quic is a good move but maybe we didn't need an official http/2 standard in the meantime.

> If I were the Internet Dictator, I'd be a big fan of standardizing the semantics of http separate from the physical transport and then defining best practices for using alternative transports in the wild, and then eventually standardizing what comes out of that process.

This is what the IETF has been doing for several years already as part of the core HTTP specification rewrite. Namely, see the HTTP Semantics document https://www.ietf.org/archive/id/draft-ietf-httpbis-semantics...

Semantics are common, how those map to specific transport - how it looks on the wire - is then the job of each individual HTTP version.

It reads like you're overlooking the act of standardization in improving the definition of a protocol. Experimentation is good but often only focuses on a limited set of use cases, deployments, or implementations. Standardization is great for interoperability across a more diverse set of all those three. This typically yields protocols that are more generally applicable for a wider population of the Internet.

Google QUIC was focused on an HTTP use case and thatbran through the entire design. Through the IETF process, the design evolved to be a far more generic transport with rich extensibility and modularity.

> is it an aggressive standardization campaign by one interested party

Everyone from FB to Microsoft to even Huawei wanted QUIC. It's was not something that only Google was interested in. Hell, the primary author left Google well before the WG was done.

> But historically RFCs were built on practice, not the other way around.

Right from the early drafts, there have been multiple independent QUIC implementations constantly being tested. There was plenty of working code before there was rough consensus.

> standardizing the semantics of http separate from the physical transport

What actual semantics of HTTP changed with QUIC?

You've got it backwards. The final problem with HTTP/2 couldn't be solved by the application layer protocol alone, so "intermingling" is a "not-even-wrong" complaint -- if you transported a large JPEG and an HTML page over the network in an HTTP/2 stream, a lost packet in the jpeg file would cause the html stream to potentially stall until it was corrected and finished, even though you can load both concurrently. This effectively causes head-of-line blocking on what should be a concurrent system. So a primary attraction of HTTP/2 is effectively defeated and possibly even made worse; it is a behavior instituted by TCP, there's nothing you can solve it with.

QUIC was "intermingled" because it exactly solved this problem at the transport layer, the only place it can be solved. They couldn't just introduce any random thing, and no other protocol existed, so they ratified QUIC independently as its own protocol and applied it to HTTP/3 at the same time. They had to introduce a new standardized transport if the hope was to solve the HOL blocking issue. If you "dis-entangled" it you'd just have HTTP/2. The new transport protocol is the defining characteristic.

Furthermore it all would have been pointless to define QUIC as a standard without actually applying it in practice, in anger, to real workloads. Which is exactly what multiple stakeholders in the process did throughout the multi-year ratification process. What are they going to do, define QUIC, sit on their thumbs for 5 years and cross their fingers hoping someone else tries it?

There are other factors like most of the involved stakeholders had previous experience watching with other attempted standards in similar veins like TCP Fast Open die on the vine. That also influenced decisions like using UDP for QUIC's transport (and the ever present discussions about middleboxes.) Actual experience indicates most low-level protocols have ossified badly or are handled very poorly, including TCP and UDP except most things can handle those, so moving the transport layer into the application stack is seen as providing actual agility needed to respond to issues like the HOL blocking issue. That's how they revised the protocol several dozen times over the years; applications are actually fairly agile. Now you could argue these people should have discovered that issue when designing HTTP/2, but "Why didn't this thing get discovered and found out?" is always a weird complaint. Why would you expect everything to be found out early? I just can't answer that complaint honestly. Weirder things have happened than "Some engineers missed something."

Many people seem to hold some unwritten belief that if a protocol is defined as, say, an application layer protocol at one point, then it can never ever redefine its parameters to be anything beyond that. It can never become a transport protocol, for example, or have the two be mandated together, despite decades of experience suggesting that might be a fruitful path for many technical reasons. But there's no random rules of the universe defining what a network protocol looks like. It can be whatever we want it to be. Humans define this, there isn't like a book of 10 commandments out there written by God.

There is also no necessary indication that HTTP 1.1 sticking around for so long versus HTTP/2 necessarily meant it was the best among all options, by the way. We abused it a lot for many things. Being around for so long could also mean we were just bad at enacting change; certainly browser monocultures have made change easier for better and worse, on that note. But the whole HTTP/2 and HTTP/3 process has been about 13 years on its own if you count it as starting with SPDY. It's not like there's 200 years of precedent for how long it should take.

Most of this is just retreading the other thread so I'm largely going to just say you can read my replies there, but to explicitly reiterate:

> They had to introduce a new standardized transport if the hope was to solve the HOL blocking issue.

I take no issue with quic being standardized and I applaud that it was.

> Furthermore it all would have been pointless to define QUIC as a standard without actually applying it in practice, in anger, to real workloads.

Protocols are deployed on real workloads in anger in practice every day without even having an ietf rfc, let alone being blessed as the next official version of one of the most important protocols on the internet, and it works out fine. I never said, meant, or in any way advocated for anyone sitting on their ass.

So based on reading your other responses your complaint is that "I think RFCs are built on practice than the other way around" which doesn't mean much, considering that's what is happening! They are building on practice. Most if not all RFCs anywhere, including the IETF, actually see lots of revisions, and QUIC is hardly very different, and there is no hard rule about how standards are formed. This is how it works.

EDIT: I was too rude/curt and re-edited this. Sorry.

Given that they have had 34 some odd drafts and continuous errors, with some still not fixed, QUIC is not confidence-inspiring. As an ordinary internet user, the problem I see is that QUIC development is being led by companies, not academics,1 that see the internet first and foremost as an advertising and commerce medium, as would be expected since they are companies pursuing business interests. The issues around those commercial uses are prioritised while issues around ordinary internet user privacy and control are secondary. Certainly QUIC is not the first2 and and will not be the only UDP-based reliable transport protocol but what will be interesting is whether firewall administrators will treat all the alternatives equally. It may be that the corporate-sponsored ones like QUIC are allowed through while ones written by academics or other non-commercial authors are not.

1. Many of the people leading development of the older HTTP versions were academics.

2. See, e.g., CurveCP: https://curvecp.org Having played around with CurveCP before QUIC existed, and having already thought CurveCP was amazing, the later commercial effort led by Google has left me underwhelmed.

https://dl.acm.org/doi/10.1145/3485983.3494865 seems like a more understandable source.

This link was much easier for me to read and parse, thank you scarming.

I would appreciate it if the industry would stop trying to shove everything into user space. Imagine it's 40 years in the future, and we still have to kludge new protocols over an 80 year old TCP/IP stack because nobody wants to add a new protocol to the standard. Every application ships with its own unique protocol stack implementation. Duplicate code abounds. Bugs are fixed haphazardly. Quality and stability varies by application. RCEs are more common because an increase in implementations increases the probability of security bugs.

Compare that to finalizing a new protocol, putting it in the kernel, and letting all applications just connect() and send() and recv(). We lose the ability to constantly mutate the protocol whenever FAANG feels like it, but we also gain the ability for every application to use the new protocol just by changing a flag passed to a syscall.

Pushing it to the kernel means you have to upgrade your kernel to use it. Instead, you can just include a library that exposes a send() and recv() style API to the main app code, while the library just throws everything into a UDP socket. Seems better to me, IMO.

The real issue is that we can't get away from UDP/TCP because of middleboxes (firewalls, load balancers, etc) and the fact that switches only know how to hash on 5 tuples of protocols they know about. If you're lucky, your router will be able to look inside a GRE header or something, but that's about it.

When was the last time you used SCTP, for example? (If you've used it recently, I bet it was tunneled over UDP)

vendors can't get away from middleboxes, because Enterprise customers want middleboxes. But if customers want a completely reinvented underlying technology, everyone is forced to either adopt it or become obsolete themselves.

This whole middlebox obsession is some kind of Stockholm syndrome. Here you are telling me there's people who make shitty devices that don't respect network traffic and because of that you can't change network traffic. It's like saying there's someone breaking into houses to steal stuff, so we can't install stained glass windows because a thief might break them.

technically speaking, switches do not hash TCP information but hash based on data in the Ethernet frame. (this is mainly used for correctly balancing load across links in an lacp bundle).

High end routers are able to do pretty conplex lookups into their asics, but this usually only extends to specific use cases. (as far as I am aware for instance, an juniper MX series is able to do a lookup of both vlan tags in a single asic pipeline step when using q-in-q. the same also applies to mpls stacked lable lookups on provider edge routers).

> technically speaking, switches do not hash TCP information but hash based on data in the Ethernet frame.

I'm not sure the distinction you're drawing here. When my switch balances based on port numbers,-- yes, it's technically just peeking at "data within the ethernet frame". (is this IP? OK, let's add the address at this offset to the hash. Oh, it's TCP too? let's add the port numbers at this offset to the hash).

L2 hashing puts everything from a single pair of hosts on the subnet on one wire, but this can suck when a lot of traffic comes in through a gateway. L3 hashing fixes the gateway problem. L4 hashing lets individual connections have entire links. And, of course, the more fine-grained the hashing, the better the balancing can be expected to be and the lower the odds that 80% of your traffic goes down 1 link in a 4 connection bundle.

The separation between a switch and a router is getting blurrier by the day, but any datacenter grade switch today will support L4 hashing for LACP. Even my el cheapo Mikrotik does.

And it would be great if SCTP support was common across many platforms and supported over networks so we could have something (in ways) better than TCP or UDP.

Unfortunately it's over 20 years old and people just shove it in user space and use it over UDP (WebRTC).

Your sentiment is appreciated but it wouldn't be a way to get anything actually done :/.

This is quite interesting. The only issue I see is when you want to stream real-time audio and video over TCPLS. That is when you want packets that are dropped, late or reordered to not interfere with the delivery of other packets. Using TCP as the underlying transport makes that problematic, and even having multiple TCP connections cannot help much with that.

interesting debate whether encryption should occur at the network vs. transport layer. on the one hand, encryption tends to have session state, so the transport layer makes sense but on the other i often feel i'd like to see a new version of IP where encryption is baked in as a primary design principle thereby eliminating all notions of leaks.

IPsec tunnel mode delivers that function https://datatracker.ietf.org/doc/html/rfc4303

This contrasts to IPsec transport mode which only encrypts the payload not the IP info https://upload.wikimedia.org/wikipedia/commons/6/64/Ipsec-es...

Generally there is encryption available at each layer in the network stack but middleware (operating systems, firewalls, NAT) tend to get in the way at lower layers. E.g. IPsec is often now run with NAT traversal which means the ESP header is stuck under a UDP header so it can get through to end devices even though IPsec itself doesn't require this. This, and a few other things, have left IPsec as a thing most people only know for corporate VPNs.

i'm envisioning more of a sort of ipvx, 64 bit addresses, encryption is non-optional and dos resistance is built-in.

It's mandatory to support in IPv6 so the only thing lacking in the general population's mind is a convincing enough reason for people to force it for every connection which will cause a lot of breakage due to the aforementioned middleboxes. But if you can convince that there's nothing about existing IPv6 implementation that would prevent it from working as described today.

Switching to IPvX would be the same problem as convincing people to always encrypt with IPv6 just amplified as there is 0 hardware with existing support for such a protocol and IPv6 has had a 20 years head start on rollout that would be wiped away. It'd have the added "bonus" of being even harder to pitch adoption for since "NAT on IPv4 is working fine" and you'd still have to wait for those people THEN the IPv6 people to migrate.

Meanwhile things like QUIC, HTTP/3, and TCPLS give all the same advantages as migrating to the hypothetical IPvX without breaking when they run into middleboxes. Just think of it as IPvX but with a wider header field than you were planning. After all you'll still need some header that tells the network where to deliver the packet then the encrypted payload it's just that that dumb destination header uses more bits than it needs to if it were designed fresh.

i think the big problem with ipv6 has been that there just hasn't been a compelling enough reason for most users to care. it's great for network operators and planners who worry about things like limited address space, but otherwise, it means nothing to end users other than more cumbersome addresses (and potentially slower routing table lookups).

i think a major design goal of something like IPvX would be to eliminate the need for any firewalls, NATs or network partitions. everything would be hardened for the public internet, every datagram would be encrypted, every listening service would be ddos resistant and this would all be handled by the networking stack by default.

crucially i think that there would have to be a hook to get end users to use it. i think there's a lot of room for next generation decentralized public internet identity, naming, reputation and commerce protocols. if these next generation services only lived on IPvX and made direct use of expanded address space, then maybe it would be compelling enough for an upgrade.

of course there are concerns about replacing all the old networking hardware and the environmental impacts of such, so maybe that could be mitigated by exploiting planned sunsets and also mark a sunset for planned sunsets.

spitballing is fun! :)

Some related discussion in this Twitter thread:

https://twitter.com/alagoutte/status/1532013841718120449?t=W...

TCP is terrible at long distance data transfer. I am sure some people will still prefer it since it can manage congestion but shouldn't newer things use UDP and handle delivery guarantees at the application layer?

There is nothing inherently wrong with modern TCP for long distance data transfer. You do need to use the right congestion algorithm and some extensions enabled to handle large window sizes to get high speeds at long distances. But UDP does not magically solve anything; if you want reliable data transfer over UDP you have to basically implement almost everything TCP does on top of it. The main benefit is that you are no longer relying on the operating system's implementation of TCP, the drawback is that you won't get any help from the OS or even hardware.

The congestion control algorithms that are applied in TCP and QUIC are actually even the same ones (e.g. Cubic, BBR, etc), so the technologies behave not too different over higher RTTs. Both will ramp up a congestion control window, which can get pretty big.

There are differences in behavior regarding what happens if loss is encountered and what exactly is retransmitted, but those still don't make QUIC radically different in terms of behavior and performance.

I thought you can't keep increasing window sizes, it is a physics limit? The mere fact that you need to wait for an ACK is the problem?

https://en.m.wikipedia.org/wiki/Bandwidth-delay_product

As GP said:

> if you want reliable data transfer over UDP you have to basically implement almost everything TCP does on top of it.

Even QUIC has an ACK-like signal so you will never avoid it as otherwise you will deal with the two generals' problem (https://en.wikipedia.org/wiki/Two_Generals%27_Problem). The reason QUIC is more efficient than TCP is simply due to the fact that TCP can't evolve further due to middleboxes (called ossification, https://en.wikipedia.org/wiki/Protocol_ossification), not that TCP is inherently bad.

Thanks for the explanation, I didn't know that about QUIC. But QUIC is trying to implement reliable delivery for applications, which makes sense as a general purpose protocol. but if I were doing a file transfer app/protocol these days for example I would send hash+Mac+metadata reliably and then check chunks of the file (unreliable+encrypted UDP)as they are decrypted (have to anyways for integrity checking as part if decryption), and peform reliable retransmission request when decryptiom fails. That's what I meant by application layer, application and use-case specific protocols. but perhaps general purpose protocols are still better (I am not sure, I was asking in my post, sorry for not being clear)

You could build this as an application mapping over QUIC. Within a single QUIC connection use reliable streams for the metadata delivery, and unreliable datagrams [1] for the payload. The benefit of a single connection is shared security context and shared fates of different datatype transfers.

[1] - https://www.rfc-editor.org/rfc/rfc9221.html

Then you would generally try to look at probabilistic reliability. One way systems like digital broadcasting requires probabilistic reliability so a lot of research was done here. Digital broadcasting has produced a lot of answers in this area, as you need to deal with both data corruption and missing data. On the data corruption side, you would want to implement a forward-error correction (FEC, https://en.wikipedia.org/wiki/Error_correction_code#Forward_...) to allow recovery of corrupted packets, while on missing data it can be mitigated by fountain codes (https://en.wikipedia.org/wiki/Fountain_code) that is basically FEC but deals with missing pieces instead of incorrect data. This of course does not guarantee 100% delivery but in practical terms this is humanly reliable.

Thank you, that certainly gives me a lot to think about in this area.

With scaling tcp can use window up to 1GB. Which is enough for 1s rtt which is ridiculously high @ 8gbps which is also ridiculous for long haul on a single flow. In practice you probably almost always just parallelize into multiple flows

I had no idea TCP can reach 8gbps over 1s rtt. I suppose folks that tune it and use the right algorithm can get that or multi-threaded transfers working. The problem is most services and applications are not specifically tuned for this.

They don't need to; it is the kernel that implements TCP, and on Linux at least you can easily change the congestion control algorithm that is used and some other parameters. Again, UDP doesn't magically solve anything, and QUIC implements congestion control in userspace, so then you have to wonder: is my QUIC implementation tuned for 8gbps over 1s rtt?

Note that your example of file transfer is a special case that an application can be specifically tuned for; it can have a receive window that is as large as the file it is going to receive, because it can use the file on disk as the backing store for that window, and it can implement an unlimited form of selective ACK. This is really not something that you can do with a generic transport protocol that implements the POSIX sockets API.

Some thoughts from someone working on QUIC for a couple of years:

There's some good ideas in this one! The idea of just keeping a TCP stream per Tcpls stream while sharing some crypto state means it can be really efficient, since all optimizations that exist for TCP (including hardware offloads) will continue to work. That is also visible in the efficiency benchmarks [1] on the site.

One might even argue that with such a design where the most expensive parts of TLS (handshakes) become no-ops an introduce no additional latency, techniques like HTTP/2 become unnecessary - one could keep just having a single TCP stream per request and enjoy better flow control, fairness and observability than with anything application managed.

However there are some challenges that need to be resolved for running protocols on "real-world infrastructure and datacenters", which change the picture a bit. One is that a single IP address announced by large-scale webservices doesn't necessarily point to a single server, but to e.g. a rack of them or even bigger units. This means that a second TCP connection might (or even should be) established to a different server - which would not be aware about the TLS session state. If the state is kept on a different server, the original one would need to forward all packets either via IP-in-IP encapsulation or by creating a separate upstream TCP connection and proxying all data bidirectionally. All those approaches would be fairly expensive - every time a TLS session would be joined the amount of packets handled by a server would be double of the ideal version.

While a similar challenge exists for QUIC, it's actually cheaper on average: Since multiple streams use the same QUIC connection and usually also the same source/destination IP and port pairs, traffic would usually be directed to the same host by common infrastructure and require no special handling. It would only require effort to "fix" misdirected QUIC connections in the event of connection migrations - which are rather rare. If packet forwarding is required, it can be achieved rather cheap and easily by interpreting the QUIC connection ID fields in all UDP packets and doing raw UDP packet forwarding.

Therefore as far as I understand the proposals so far, it might not be necessarily clear-cut on whether running QUIC or TCPLS would be cheaper and an ingress or load-balancing layer - which are the layers where both protocols have the biggest benefits due to the highest amount of concurrent connections and higher client-facing RTT.

But maybe the challenge for TCPLS could be somehow solved by having a shared place for the TLS state between hosts and making sure streams are operating fully independently - but it requires a bit more thoughts.

I'm wondering how close such a proposal is then to the existing TLS session resumption mechanisms - just extended by allowing more than 1 follow-up session.

[1] Note that the diagrams are really about efficiency and not performance. They tell nothing about what throughput might be available for end-users that are e.g. 100ms away from the server. For local tests the delay and packet introduced by large-scale internet infrastructure doesn't play a role.

Very interesting thoughts, thank you for sharing!

Lots of dupes:

https://news.ycombinator.com/item?id=31589575

https://news.ycombinator.com/item?id=31569066

They need to have some discussion to count as HN-dupes and those don't.

That seems a little wrong - I can just spam uninteresting, yet topical things to HN, and no negative consequences accrue? People will see the headline at least, maybe click on the article. I get the clicks every time, and I get to maybe repeat a Big Lie over and over, which will help my goal of framing the discussion my way.

But thanks for the correction! I appreciate it.

There are other mechanisms that handle things like spamming and even copycat links.