The way I look at it a lot of what we logically think of as the network layer often exists in userspace anyhow. That's the point of DPDK/snabb/netmap and other kinds of driver bypass.

The important design distinction about the layers is what element has access to what data. (e.g. routers need to see IP addresses to do their job, port numbers help kernels segment permission models, etc..) The rest of it is just about logical models, real-world workarounds, and luck...

HTTP/3 will be able to get high bandwidth, buttery responsive restarts to connections far too long idle to keep "open" because it integrates security, application, and transport. That's thoughtful design, not a workaround.

I'm going to paraphrase (and maybe bungle, because I don't have it at hand) from my favorite networking book of all time - the underappreciated _Network Algorithmics_ by George Varghese. He describes layers as a lovely way to model and think about a protocol or design, but often a terrible way to build one. I've spent a lot of time thinking about that, and I think QUIC gets it right - the layers are clear in how they inter-relate but they do so without being independent.

The article mentions gives an example of why it’s difficult to improve TCP further.

TCP Fast Open was standardized 8 years ago and is barely used. This is because updating TCP requires kernel updates, which just isn’t going to happen on most mobile devices.

Thus, moving the protocol to userspace makes a lot of sense.

> Thus, moving the protocol to userspace makes a lot of sense.

Raw IP sockets are accessible from the same userspace facing APIs as e.g. UDP sockets and don't require climbing up the stack. Unfortunately operating systems started to consider custom protocol implementations security risks but rather than reverse that thinking we've just continued to abstract up past it.

In reality I think "where it is implemented in code" was a small portion of QUICs design choices compared to "IPv4 NAT & external firewalling has ossified protocols" which is a similar story of "just abstact up to avoid the issues". Unfortunately in that case I don't think abstracting up isn't as permanent a solution as it was on the OS side.

Raw sockets don't really allow for multiple applications to use the same custom protocol. If, for example, chrome and firefox were both running, which gets packets destined for the QUIC transport protocol? The kernel wouldn't know; without the UDP header it can't distinguish flows.

Likewise NAT devices typically support UDP flows today due to their prevalence in games, but if you introduce a new transport protocol at the IP layer, they wouldn't be able to identify which flow (and therefore which NATed endpoint) the packet is destined for.

> Raw sockets don't really allow for multiple applications to use the same custom protocol. If, for example, chrome and firefox were both running, which gets packets destined for the QUIC transport protocol? The kernel wouldn't know; without the UDP header it can't distinguish flows.

In reality raw sockets work in a way that the question is the reverse of what you describe. The kernel will check 2 things: - Which raw sockets are bound to the protocol number seen in the packet - Which raw sockets have issued "connect" to the sending IP

Any and all raw sockets that match these will receive the packets. In such a sense the protocol (QUIC) needs to have some way to identify streams so that if e.g. both Chrome and Firefox browse to the same server they don't interfere with each other. QUIC innately has this functionality due to the way it implements encryption. Ideally the OS would allow a raw socket to register something akin to a BPF filter though as that would make it equally as efficient as UDP socket tracking even in the edge cases.

> Likewise NAT devices typically support UDP flows today due to their prevalence in games, but if you introduce a new transport protocol at the IP layer, they wouldn't be able to identify which flow (and therefore which NATed endpoint) the packet is destined for.

This is actually what I was referring to when I said:

> "IPv4 NAT & external firewalling has ossified protocols" which is a similar story of "just abstact up to avoid the issues"

We continue to make non choices to build up the stack rather than implement systems that are interchangeable.

Chrome and Firefox could develop standardized system service which will deliver package to a proper application. NAT is not needed in a bright IPv6 world of the future.

Though I don't know what's wrong with UDP. 8 bytes of overhead for 1450 bytes IP payload is 0.5% bandwidth. Checksum overhead should be negligible.

> system service which will deliver package to a proper application

That's expensive. Even if you avoid copying the packages by sharing memory between processes, there are still a lot of context switches...

I think TCP fast open is a bad example for this. None of the common socket libraries that I know (never mind http libraries) have gained support for TCP fast open yet.

It's a bad example because it's badly adopted?

It's a bad example of middle-boxes causing ossification, as the adoption has been more limited by library/framework support than middle-boxes blocking it.

Chicken and egg problem. It doesn't tend to work reliably because of middle boxes so there is no push to implement it widely. It is not widely implemented so there is little pressure to update middle boxes.

That doesn't sound like a particularly convincing reason. In order for mobile devices to benefit from HTTP/3, commonly used HTTP client libraries will have to be updated. Which usually happens on a similar timescale as kernel updates anyway.

Much easier to update an app and its HTTP library than mobile device kernels. Particularly given how a large proportion of mobile devices are unsupported and won't get kernel upgrades any more.

The thing is, there are two possible extremes:

1. Our design is complete, error-free and designed to stand the tests of time. It will be in common use, largely unchanged, in 40 years time. The TCP of the 2010s. There is widespread industry support. We want to move to the application layer to ease the initial roll-out.

2. Our design will need to change every few years, even we authors don't think it's finished. This is a Google-only project and most vendors are refusing to support it as they think it's badly designed crap. The ActiveX of the 2010s. We want to move to the application layer so we can force it through without anyone else's support.

Where are we on the spectrum between those two options? I don't know.

Great question!

An app can control which HTTP library it’s using and even bypass the built in one on the mobile device. That’s not possible in the case of TCP.

So no, it’s not the same.

Another issue they mention in the article is middleboxes, which basically will never be upgraded, and will never support new TCP features.

Nor will they support new HTTP-features.

Some people implement network stack in user-space. Abstractions are good, but they incur performance penalty or other restrictions and sometimes you need to remove those abstractions. I guess, web is too important today, so optimizing it might worth it, even if that requires unconventional measures.

Is this not a valid approach then? The issues of ossification and strict allowance for just known protocols appear to be big enough to cause things like SCTP to not have a viable, widespread use in their future.

I believe that we will see ossification of QUIC eventually too. TCP has been around for decades, anything around that long is going to have issues rolling out new changes in a backwards compatible way. TLS 1.3 and the lengths it had to go to with backwards compatibility with middleboxes is another good example.

I hope that QUIC has used these lessons from TCP and TLS to make changes in the future as easy and effective as possible, but I’m sure it’ll still have its limits.

The IETF QUIC working is well-aware of this, and is attempting to save design room for future QUIC versions as much as possible. The "QUIC invariants" spec documents everything that is guaranteed not to change, but other than than everything in a future QUICv2 could be updated (e.g. tls version, features, large parts of packet header layout).

I fear that unless they are already actively using different values for those updatable values that middleboxes will implement a de facto version of QUIC that “just works” with what is out now, but no regard to gracefully handling forwards-compatibility. Example is the SSL version field which has become static because many implementations didn’t handle an unknown value gracefully.

It runs over UDP so they did go back to the protocol layer when designing a solution. Thats the whole point is to fix the tcp halt and retransmit delay when a packet gets lost.

So it will be firewalled by most corporate firewalls then?

Hardly sounds useful.

Why would it get firewalled? DNS runs over UDP on port 53

It is not necessarily bad. Moving complex stuff to user space can be good sometimes. There are things this would break like splice but also allows much tighter integration with applications because you can get and set state without system calls. As you layer on more complex app level adaptations to connection failures having it in user space means the client has the same capabilities irregardless of platform which is important so that both client and server can make mutual complimentary adaptations to the same conditions. This stuff is not really needed for web, it is mostly video conferencing type stuff that gets the biggest benefit. Googs new game service for instance.