News for hackers. Stuff that matters.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by throwaway892238 on Sunday October 30, 2022 @11:57PM (#33401882)
Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo.

What's kind of hilarious about this paper is, these are just the network-layer problems! It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP as some sort of universal tunnel encapsulation for all application-layer protocols, rather than a hypertext locator. And then there's all the non-backend problems!

And that's just TCP. We still lack any way to communicate up and down the stack of an entire transaction, for example for debugging purposes. We should have a way to forward every single layer of the stack across each hop, and return back each layer of the stack, so that we can programmatically determine the exact causes of network issues, automatically diagnose them, and inform the user how to solve them. But right now, you need a human being to jump onto the user's computer and fire up an assortment of random tools in mystical combinations and use human intuition to divine what's going on, like a god damn Networking Gandalf. And we've been doing it this way for 40+ years.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by bigDinosaur on Monday October 31, 2022 @12:17AM (#33401994)
Your ideas are interesting, can you link to or explain a concrete example though? The idea of everything magically debugging itself doesn't apply to a single piece of software I've ever seen, so I'm curious what kind of design would lead to that being possible.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by Areading314 on Monday October 31, 2022 @12:24AM (#33402049)
Heres an example of an improvement to sending large files over long distances -- Tsunami protocol. It tries to get a best of both worlds to limit the detrimental effect of synchronous roundtrips in the TCP protocol for file transfers:

https://tsunami-udp.sourceforge.net/

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by fragmede on Monday October 31, 2022 @12:06AM (#33401932)
What is QUIC in your book?

Given, say, $50 million of dev time, what would you go about fixing? And in what way?

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by Areading314 on Monday October 31, 2022 @12:25AM (#33402054)
Doesn't QUIC still run over TCP? I thought it was a replacement for HTTP not TCP (Edit: looks like it replaces TCP and HTTP)
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by stevewatson301 on Monday October 31, 2022 @12:28AM (#33402088)
QUIC runs over UDP, and provides streams and encryption. HTTP/3 is designed to take advantage of QUIC streams (replacing HTTP/2 streams which were problematic due to TCP head of line blocking).

The RFCs are a bit elaborate so folks interested might want to look at this instead[1], which has one of the RFC authors explaining the basics of QUIC and HTTP/3.

[1] https://www.youtube.com/watch?v=cdb7M37o9sU

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by jhardy54 on Monday October 31, 2022 @12:28AM (#33402086)
Nope, UDP.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by klabb3 on Monday October 31, 2022 @03:22AM (#33402890)
It replaces TCP+TLS, and runs multiple streams on the same conn, supports transition from eg wifi to ethernet on at least one of the nodes. And since it's over udp implementations are mostly in user space. Which is good if you want it now, but not great for performance. Ip packets are very small so you gotta have either kernel support for quic or batch IO, otherwise it's often CPU limited (yes, really). In addition congestion control is wonky, unfortunately. In my experience (quic-go), it's too shy in the presence of TCP streams, which ends up getting more bandwidth. But that depends on the algorithm used, implementation and God knows what else.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by notpushkin on Monday October 31, 2022 @05:45AM (#33403676)
I guess you were thinking about another clever name protocol, SPDY :-)

SPDY → HTTP/2

QUIC → HTTP/3

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by OmarAssadi on Monday October 31, 2022 @12:46AM (#33402179)
In addition to QUIC, KCP [1] is another reliable low-latency protocol that sits on top of UDP that might be interesting. And unlike RFC 9000/9001 (QUIC), encryption is optional. I haven't really seen it mentioned much outside of primarily China-focused projects, like V2Ray [2], but there is also some English information in their Git repo [2].

[1]: <https://github.com/skywind3000/kcp>

[2]: <https://www.v2fly.org/en_US/>

[3]: <https://github.com/skywind3000/kcp/blob/master/README.en.md>

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by xfs on Monday October 31, 2022 @12:56AM (#33402226)
KCP uses a brute force congestion control algorithm that is unfair and inefficient. It is also poorly specified, which is probably why it is less commonly used outside circumvention circles.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by ronsor on Monday October 31, 2022 @01:40AM (#33402419)
KCP is notably used by the popular mobile game Genshin Impact.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by fomine3 on Monday October 31, 2022 @02:13AM (#33402570)
Still is looks interesting for some use cases, even though it's not fair if it's fully utilized on the internet.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by Luker88 on Monday October 31, 2022 @05:14AM (#33403485)
IMHO QUIC is nice, but a disappointment, since it could have been so much more.

Does not handle unreliable messages, still only (multi)streaming, no direct support for multicast, 0-rtt which need a lot of stuff to be manually done TheRightWay or risk amplification attacks, the (imho) under-researched (and removed) forward error correction, and more.

I just restarted working on what I consider to be the solution to this, federated authentication and a bit more, but $50M is too far to be even a dream since I am not google.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by ajross on Monday October 31, 2022 @12:21AM (#33402028)
Surely most of your ideas are already being deployed in QUIC/HTTP3. It just happens inside a UDP datagram, for compatibility. Really you're not going to see any new IP protocol layers, there's too much quirky hardware on the network that wouldn't be able to handle it. If we can't even get IPv6 to work all the way to the client, we're never seeing new values for the protocol byte.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by vlovich123 on Monday October 31, 2022 @12:27AM (#33402079)
Don’t the hyperscaled cloud providers run totally segmented networks? What’s stopping them from using something proprietary internally and just exposing TCP at the end for termination of client connections?
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by wmf on Monday October 31, 2022 @12:48AM (#33402187)
Google already does that.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by vlovich123 on Monday October 31, 2022 @01:14AM (#33402309)
I’m not aware of them using something other than TCP internally (I’m sure by now they’ve migrated to QUIC but I’m not sure that QUIC necessarily solves some of the scaling challenges / optimizes for gRPC and low latency).
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by uluyol on Monday October 31, 2022 @01:26AM (#33402364)
Google is using remote memory accesses rather than TCP for at least some classes of traffic (e.g. a caching system). They've been publishing details about how it all works too.

Also, they have a transport (Pony express) developed specifically for RPCs, rather than byte streams or datagrams.

Links: https://research.google/pubs/pub51341/, https://research.google/pubs/pub50590/, https://research.google/pubs/pub48630/, more generally https://research.google/pubs/?area=networking

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by Phelinofist on Monday October 31, 2022 @07:01AM (#33404159)
Can someone ELI5 how remote memory access works?
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by vlovich123 on Wednesday November 02, 2022 @12:25AM (#33431425)
I could be wrong but I believe they have a unified address space. There’s dedicated hardware that then owns a given memory range. On an access it will fetch it from the remote location matching that address on demand and store it in real memory in space allocated to it. Presumably it evicts stuff if there’s insufficient memory. Once the memory is brought over either a virtual address range is remapped to point to main memory or the ASIC just has a TLB itself.

This is pure speculation based on seeing the word ASIC in one of the summaries but it seems like it could be reasonable.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by kccqzy on Monday October 31, 2022 @02:10AM (#33402555)
I don't think Google-internal communications happen over gRPC. Maybe the protocol was design with an ambition to replace their internal RPC system but it probably failed at that.

They have a new system called Snap although judging from the paper I don't think it can completely replace TCP: https://research.google/pubs/pub48630/ My understanding is that Snap enables new use cases including moving functionality previously done via RPCs to RDMA-like one-sided operations. I think it is complement to RPCs but does not replace it.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by ajb on Monday October 31, 2022 @01:16AM (#33402318)
They do, it's called DCTCP. Although it's actually an open standard.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by alexgartrell on Monday October 31, 2022 @01:03AM (#33402253)
> corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo.

This is ridiculous.

Hyperscalars see an immediate ROI from efficiency/reliability improvements and actively invest in TCP alternatives all of the time. It's just really hard.

Networking companies see an ability to differentiate their products from their peers and work on this kind of thing as well. I did a 3 second google for "QUIC acceleration Mellanox" and got a hit on Nvidia's blog right away.

You just can't trivially replace something with an investment totally 50 years of clock time and thousands of years of engineer time. It will either take a long time or a massive shift in needs/technology. FWIW, I wouldn't be surprised if the high-performance RDMA networks being put together for AI workloads were the thing that grew into the "next" thing.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by oconnor663 on Monday October 31, 2022 @01:08AM (#33402273)
> 50 years of clock time and thousands of years of engineer time

It's not just the size of the investment, it's that it's the protocol everyone uses to talk to other people's machines, and you can't upgrade or replace other people's machines.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by kebman on Monday October 31, 2022 @01:46AM (#33402455)
Yes you can. Just offer a better product, and people will buy it instead of the old or bad product. Better yet, make the new product backwards compatible, and fewer people will have qualms about forking out for it. Better yet, do an aggressive takeover, like Microsoft did, and just force the entire industry to adopt your stuff...
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by scantydolt on Monday October 31, 2022 @02:09AM (#33402552)
Great! When do you think you'll have it done?
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by kebman on Monday October 31, 2022 @01:29PM (#33409365)
Done? What do you mean "done"? Consulting hours are much better on projects that cannot ever be finished!1
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by atoav on Monday October 31, 2022 @02:42AM (#33402690)
You mean like IPv6?
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by sneak on Monday October 31, 2022 @10:23AM (#33406119)
I think QUIC/http2 is a much better example.

Google made that happen almost unilaterally via their Chrome dominance.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by withinboredom on Monday October 31, 2022 @12:43PM (#33408452)
I mean, this is how new features come about, for the most part (look at ajax, from Microsoft's IE dominance). The consortium allows anyone to contribute, not just the dominate browser, but the dominate browser will always be able to experiment with new web features without having to discuss it with anyone.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by alexgartrell on Monday October 31, 2022 @02:45AM (#33402702)
In this case we're talking about within the Datacenter, and you could conceivably update every network device and system to talk the new thing if you wanted. This is more achievable at a hyperscalar, where there tends to be < 3 distinct protocols, proxies, etc.

TCP gives you three things: 1. Reasonable performance - This is hard but not impossible to replicate 2. Reliability - This is very hard to replicate because networking edge cases are very hard to isolate 3. Fairness - this one is roughly impossible, because the "fairness" is an artifact of the experimentation and tweaking of Congestion Control Algorithms.

To elaborate on fairness, dynamic traffic control of all flows within a DC while maintaining high utilization is roughly impossible. You can get really close to this by picking your battles wisely (i.e. solid demand control for data warehouse workloads), but you'll always end up counting on individual flows to react appropriately to loss. They need to back off enough to make room for others without tanking their own throughput.

The people who design and implement these algorithms are definitely geniuses, but even they rely on TONS of empirical evidence to narrow parameters to what's appropriate. Of the Kernel Networking people I've worked with, Lawrence Brakmo had the most sophisticated network testing harness I've seen. Even then, you don't really know if it works (and can't finish tuning it) until you run it in production.

Running novel congestion control algorithms in production at a sufficient scale to figure out whether or not they're working appropriately is a great way to kill your network, so we end up conducting the equivalent of CCA drug testing to roll it out slowly and safely.

The end result of all of this is that it's really hard to solve the "arbitrary connections sharing arbitrary network topologies with high utilization" problem quickly enough for it ever to look like a breakthrough rather than just steady progress.

It's also worth noting that it's usually easiest to prove performance, so you'll see a lot of excitement about performance benchmarks from people who don't yet know what they're about to learn about networking. We were very much in this camp at Facebook when we were all-in on memcache-over-udp, and we later abandoned it completely.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by lamontcg on Monday October 31, 2022 @03:16AM (#33402854)
After having lived through Amazon's early (pre-2003ish) UDP-based networking I got a laugh around 2006-ish or so reading about how facebook was into UDP. I assume there are people who worked there who still have the scars.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by tezza on Monday October 31, 2022 @03:56AM (#33403097)
Do you have any specific problems you can elaborate with the UDP ?

UDP used successfully many places.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by adamcharnock on Monday October 31, 2022 @04:42AM (#33403328)
I’m really looking forwarding to seeing the original commenters reply on this. But I’ll share my experience too.

I’ve found UDP to be great for latency but pretty awful for throughout. Especially over longer routes (ie inter-region transports). Also, if you fire UDP packets out of a machine in a tight loop then there is every chance you could overload various buffers and just loose them (depending on the networking hardware).

TCP is comparatively amazing for throughput, but you do take a latency hit (especially on the initial handshake, which doesn’t exist for UDP).

There are some very experienced people commenting here though, and I’d be happy to be corrected or expanded upon.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by Tor3 on Monday October 31, 2022 @06:29AM (#33403956)
Anecdotal, but I've some experience in running both TCP- and UDP-based VPN over long-latency links (I worked from half around the globe for some years).

With OpenVPN it's easy enough to test - configure for UDP, or configure for TCP. With long latency, and a tiny amount of packet losses, running TCP over TCP OpenVPN completely stalls, while TCP over UDP OpenVPN is excellent - it's around the same performance as running direct TCP, or sometimes actually better. At work we've also used other types of VPN setups (for engineers on the road), and the TCP based ones (we've used several) work fine most of the time, but if you try that from far away it becomes nearly unusable while UDP OpenVPN continues to work basically just fine.

The TCP over TCP VPN performance problem (over long latency links) presumably has to do with windowing and ack/nak on top of windowing with ack/nak.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by ay on Monday October 31, 2022 @08:55AM (#33405072)
The TCP over TCP performance problem can be summarized as follows:

Because the underlay TCP is lossless (being TCP), every time the overlay TCP has to retransmit, it adds to the queue of things that the underlay TCP has to retransmit (and the need to retransmit happens more or less at the same time).

So instead of linear increase in the number of packets, you get ~quadratic.

This balloons the required throughput needed to “rectify” the issue from the protocol standpoint at both levels - usually precisely at the point when there’s not enough capacity in the first place (the packet loss is supposed to signal congestion).

If you are very lucky, the link recovers fast enough that this ballooning is small enough to be absorbed by the newly available capacity.

If the outage is long enough, the rate of build-up of retransmits exceeds the capacity of the network to send them out - so it never recovers.

Needless to say, the issue is worse with large window in overlay TCP session - e.g. a sudden connectivity blip in the middle of the file transfer.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by jstimpfle on Monday October 31, 2022 @06:58AM (#33404137)
UDP is little more than IP, so there isn't a technical reason why UDP couldn't be just as fast as TCP _per se_. But from when I was toying with writing a stream abstraction on top of UDP in Linux userspace, I came to the same conclusion, it's hard to achieve high throughput.

My guess is that this is in part because achieving high throughput on IP is hard and in part because it's never going to be super efficient at this level (in userspace, on top of kernel infrastructure that might not be as optimized towards throughput like it is in the case of TCP).

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by stevewatson301 on Monday October 31, 2022 @07:21AM (#33404291)
You can use eBPF/DPDK these days for hardware offload.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by nextaccountic on Monday October 31, 2022 @08:56AM (#33405089)
What about QUIC? Do you think that HTTP/3 will suffer from throughput as well?
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by Bluecobra on Monday October 31, 2022 @09:10AM (#33405228)
> I’ve found UDP to be great for latency but pretty awful for throughout.

UDP/multicast can provide excellent throughput. It's the de facto standard for market data on all major financial exchanges. For example, the OPRA feed (which is a consolidated market data feed of all options trading) can easily burst to ~17Gbps. Typically there is a "A" feed and a "B" feed for redundancy. Now you're talking about ~34Gbps of data entering your network for this particular feed.

Also, when network engineers do stress testing with iperf we typically use UDP to avoid issues with TCP/overhead.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by kazen44 on Monday October 31, 2022 @03:10PM (#33411122)
Multicast throughput is hard to measure because it is... well, multicast.

Depending on where your RP's are, and how you are transmitting multicast packets across a core, multicast performance can vary a lot.

The main advantage of multicast however, is that throughput between RP's doesn't need to be very large..

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by Bluecobra on Monday October 31, 2022 @08:28PM (#33415369)
It’s actually pretty easy to monitor the throughout with the right tools. The network capture appliance I use can measure microbursts at 1ms time intervals. With low latency/cut through switches there are limited buffers by design. You are certain to drop packets if you are trying to subscribe to a feed that can burst to 17Gbps on a 10Gbps port.

Market data typically comes from the same RP per exchange in most cases. Some exchanges split them by product type. Typically there’s one or two ingress points (two for redundancy) into your network at a cross connect in a data center.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by tiberious726 on Wednesday November 02, 2022 @04:14AM (#33432621)
Have you tried to get inline-timestamping going on those fancy modern NICs that support PPT? Orders of magnitude cheaper than new ingress ports on that appliance whose name starts with a "C", also _really_ cool to have more perspectives on the network than switch monitor sessions.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by adamcharnock on Monday October 31, 2022 @05:55PM (#33413837)
That’s interesting. And I’m sure they have some very knowledgable people working for them who may(/will) know things I don’t.

That being said, it wouldn’t surprise me if they were pushing 17G of UDP on 100G transports. Probably with some pretty high-end/expensive network hardware with huge buffers. I.e you can do it if you’ve got the money, but I bet TCP would still have better raw throughput.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by Bluecobra on Monday October 31, 2022 @08:41PM (#33415472)
Yep, 100G switches are common nowadays since the cost has come down so much, and you can easily carve a port to 4x10G, 4x25G, and 40G. In financial trading you tend to avoid switches with huge buffer as that comes to a huge cost in latency. For example, 2 megabytes of buffer is 1.68ms if latency on a 10G switch which is an eon in trading. Most opt for cut-through switches with shallow buffers measured in 100s of nanoseconds. If you want to get really crazy there are L1 switches that can do 5ns.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by adamcharnock on Tuesday November 01, 2022 @06:56PM (#33428889)
That is a really good point that I hadn’t considered. Presumably this comes at the risk of dropped packets if the upstream link becomes saturated? Does one just size the links accordingly to avoid that?
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by tiberious726 on Wednesday November 02, 2022 @03:52AM (#33432482)
Basically yes, but the links themselves are controlled by the exchanges (and tied in to your general contact for market access).

In general UDP is not a problem in the space because of overprovisioning. Think "algorithms are for people who don't know how to buy more RAM", but with a finicial industry budget behind it.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by tiberious726 on Wednesday November 02, 2022 @04:00AM (#33432533)
Do the vendors actually convince anyone to buy those hubs rebranded as "ultra high performance 5ns L1 switches"?
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by midasuni on Monday October 31, 2022 @04:27AM (#33403255)
UDP is just a protocol. I’ve served millions - even billions - of people with UDP media delivery. I use it all the time for all my work communication (WireGuard)

I wouldn’t use it to ping my gateway though, or to join a multicast group, nor would I use it to establish my bgp session, I use icmp, igmp and tcp for that.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by lamontcg on Monday October 31, 2022 @01:52PM (#33409742)
Amazon used UDP over multicast for request/response when sometimes the responses would be very large and implemented reliability on top of that through fall back to UDP unicast. This was all using Tibco RVD (taken from Bezos experience in Finance on the East Coast before Amazon I think).

The really key point there is probably the size of the responses, it wasn't just tiny atomic bits of stock information.

At one point as a system engineer I actually had to bump up the size of UDP socket data that the kernel would allowed to be sent across the entire production set of servers. SWEs were really hammering on UDP hard (the platform framework was sort of "sold" as being better than TCP though which doesn't have those kinds of limits).

The result was that one Christmas the traffic scaled up to the point that the switch buffers were routinely overflowing all the time. There was no slow start in UDP so the large payloads the SWEs were sending would go out as fast as the NICs could send them, which resulted in filling up packet buffers in the 6509s (Sup 720s I think at the time? Whatever it was the network engineers had already upgraded to whatever was Cisco's latest and greatest at the time and had tuned the switch buffers).

What made it even more fun was that as packets were dropped on the multicast routes the unicast replies created a bit of a bandwidth-amplification-attack. Then eventually the switch buffers started dropping IGMP packets, and if you drop enough of those in a row then IGMP sniffing fails and the multicast routes themselves start getting torn down. Now you get "packet loss" on one of the destination nodes which is complete. Then when it eventually rejoins it has fallen far behind all the peers (causing a bunch of issues when it was out of synch to begin with though) and then it requests more unicast messages to get caught up, creating even more of a flood of rapidly-sent UDP.

What I wound up doing is writing scripts to log into all the core switches and dump out the multicast tables and convert the IGMP snooped routes into static routes and reapply them. That let the multicast network grow as the site had to scale for Christmas, but kept all the routes in place and avoided the IGMP route flapping.

But even with that band-aid it still didn't work well and there was still high congestion and packet loss across the core switches. There were also problems with the CPU on the switches and Amazon drafted an extension to how multicast routing was done and got Cisco to implement it ("S,* routing" IDK if that's right its been 20 years). And it was a good job that the Network Engineers had ripped out spanning tree and gone L3 entirely since the packet loss and CPU congestion would have caused spanning tree to flap which would have amplified all the congestion issues. Eventually Tibco RVD was ripped out and a TCP-based gossip-based-clustering protocol was put into place.

So if you use UDP based stuff the datapackets need to be small, or else you need to throttle the senders somehow, and you need to not care about reliability. For stock ticker information it might work well, and for multimedia streaming where the protocol layer above it does slow start and congestion control. I suspect that if you dug up the network engineer responsible for those networks though that they could tell you stories about packet loss. If UDP works well at your company my suspicion is that you've either got a protocol sitting on top of UDP which implements at least half of what TCP offers, and/or you've got an overworked network engineer trying to keep it all together, and/or you just haven't scaled enough yet. I also wouldn't be too surprised if some wall street firms have switched to RDMA-over-Infiniband or something like that with link-layer and end-to-end based flow control (as this paper points out, though, RMDA has issues itself and doesn't meet all the criteria for a TCP-replacement, but that would at least stop the packet loss issues due to buffers overflowing).

QUIC is a good example of what you need to do in order to use UDP (Section 4 of RFC 9000 is all about Flow Control to prevent fast senders from DoS'ing your network switches). But for the average HN/reddit reader who reads something about how TCP is awful and has the "showerthought" of wondering about why everyone doesn't just switch to UDP in the datacenter, they're missing a massive problem in that Ethernet has no flow control and just promiscuously drops packets everywhere, so if you thoughtless slap UDP on top of that your datacenter will absolutely have a meltdown. You need to use something like QUIC at a minimum.

And buried in what I wrote above is an observation that UDP multicast doesn't really solve reliable delivery across multiple servers and failover of streams that you'd like to be able to see, that's another solution which is simple and wrong (and which it looks like Homa is trying to address).

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by nsteel on Sunday November 06, 2022 @04:19PM (#33497305)
> Ethernet has no flow control

Isn't this what pause frames and Pfc are for? (Honest question)

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by nextstepguy on Monday October 31, 2022 @06:26AM (#33403934)
Multicast storms happened regularly back in 2004
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by Bluecobra on Monday October 31, 2022 @09:16AM (#33405280)
True, there were tons of crappy hardware still in production at that time. The first job I had out of college consisted of crappy 3Com hubs (not switches) so something like Norton Ghost could take down the whole network since multicast would get flooded everywhere. Nowadays this is a less of a problem as hubs are long gone and most switches have IGMP snooping by default and would only forward mutlicast frames that someone wants.

A bad client can still cause problems though, like sending a high rate of multicast packets with a TTL of 1.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by lamontcg on Monday October 31, 2022 @09:02PM (#33415677)
Amazon was definitely not run off of crappy 3com hubs, not even back then.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by oconnor663 on Monday October 31, 2022 @01:56PM (#33409793)
> In this case we're talking about within the Datacenter

Oh gotcha. It's right there in the title, but missed it somehow :p

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by samgaw on Monday October 31, 2022 @05:28AM (#33403577)
> FWIW, I wouldn't be surprised if the high-performance RDMA networks being put together for AI workloads were the thing that grew into the "next" thing.

Maybe we were just early in giving (HFT) customers RDMA back in ~2007[1][2] but I don't see it entering the mainstream anytime soon. And after a relatively short 20 years of adoption, the "next" thing for hyperscalers is not going to be the next thing for everyone else.

[1] https://downloads.openfabrics.org/Media/IB_LowLatencyForum_2...

[2] https://www.thetradenews.com/wombat-and-voltaire-break-milli...

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by pclmulqdq on Monday October 31, 2022 @10:00AM (#33405787)
HFT networks are also a lot smaller than hyperscaler datacenters, and designed with more cross-sectional bandwidth. A good chunk of the traffic (trading-related messages) also tends to not use congestion control.

In large web company datacenters, RDMA and RoCE have had a much "rockier" path forward.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by peter_retief on Monday October 31, 2022 @01:23AM (#33402352)
At last hopefully there is light at the end of the tunnel. Big question for me is who is going to build it?
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by bsder on Monday October 31, 2022 @01:43AM (#33402439)
> corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo.

And yet every time hardware designers get the chance they redesign Ethernet and IPv4--poorly.

See: HDMI 2.0+, USB 3.0+, Thunderbolt 3.0+, etc.

My suspicion is that this paper works fine beween pairs of peers and immediately goes straight to hell after that. It is extremely suspicious that there is zero mention of SCTP and only compares to TCP and not UDP.

The problem with RPC is that multiple organizations must agree on meaning. And that's just not going to fly. It is damn near a miracle that a huge number of institutions all agree on the Ethernet/IP command "Please take this bag of bytes closer to the machine named: <string of bytes>."

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by idlehand on Monday October 31, 2022 @03:31AM (#33402929)
Never thought about that before. Ethernet supports extremely high levels of data transmission. ISB C for intrgrated charging ND data transfer makes sense, but why are there HDMI cables?
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by GTP on Monday October 31, 2022 @04:59AM (#33403414)
I think it would be an overkill to use a networking protocol to connect exactly two devices. Plus if you have something specific to video stream transfer you could maybe do some optimization specific to that use case, although I can't think of any at the moment.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by zrail on Monday October 31, 2022 @09:11AM (#33405235)
Excepting HDMI the parents examples are all networks with more than one peer. Thunderbolt and USB3 can both have arbitrary trees of nodes.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by GTP on Tuesday November 01, 2022 @06:14AM (#33419163)
The parent comment was specifically wondering about HDMI, the other examples are given saying that those have some reason to exist, while he didn't see any reason for having HDMI instead of recycling some other protocol.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by dooglius on Monday October 31, 2022 @06:31AM (#33403969)
Why do you say that these protocols are worse than Ethernet/IPv4? I'm not intimately familiar with any at L2/L3, but I don't think any have hacks as bad as ARP. (USB does have some weirdness at L1 though I know.)
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by amluto on Monday October 31, 2022 @07:20AM (#33404285)
I’ve generally considered IPv6 neighbor discovery to be a worse hack than ARP. ARP is a straightforward, fairly clean hack to layer the IPv4 addressing scheme over Ethernet, and it doesn’t pollute IPv4 itself. Neighbor discovery layers IPv6 on top of pseudo-IPv6, where the latter operates without knowledge of MAC addresses but nonetheless hardcodes knowledge of Ethernet. But hey, it eliminated the use of Ethernet broadcast in favor of a more complex but functionally identical multicast scheme.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by dooglius on Monday October 31, 2022 @08:10AM (#33404651)
Oh sure, the point is more that Ethernet/IP has to coordinate two separate ID spaces at all, whereas AFAIK no other packet-based protocol like the ones mentioned does this, so in that sense those protocols are better.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by bsder on Monday October 31, 2022 @05:33PM (#33413546)
> Why do you say that these protocols are worse than Ethernet/IPv4?

Here's an example: I connected my nice expensive audio interface to my Thunderbolt port. It worked great! Then I moved a window on my monitor and all hell broke loose. In spite of the fact that it had way more than enough bandwidth to handle everything.

See, Thunderbolt doesn't have the ability to say "This tiny packet going to there needs priority and you need to break up those giant display packets."

Ethernet has solved problems like these in standards. They're not always implemented on particular chipsets, but they exist, and you generally can buy a product that has them.

Everything Ethernet has done and standardized has generally been for a reason. If you don't implement Ethernet, then you are starting over from scratch and will have to reimplement all of that stuff.

And you're probably not smarter than the guys who did it for Ethernet.

(If I'm being charitable: what's happened is that lot of standards tried to be more cost optimized than Ethernet. The problem is that transistor prices keep coming down. Eventually the price delta between Ethernet and <whatever> becomes inconsequential and you're basically left with real Ethernet and "kinda crappy" Ethernet at almost the same price.)

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by duped on Monday October 31, 2022 @01:57AM (#33402499)
Every major corporation has multiple research organizations doing nothing but invest in things that don't have immediate shareholder value.

What you're talking about though isn't just coming up with new ideas or even new products. It's replacing hundreds of billions in infrastructure wholesale. The scale at which these changes needs to happen to be practical are at the cluster level in a single data center. If you can propose something that fits that bill there are a few companies willing to pay you millions in salary as an engineering fellow to do it.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by simplotek on Monday October 31, 2022 @03:40AM (#33402995)
> What you're talking about though isn't just coming up with new ideas or even new products. It's replacing hundreds of billions in infrastructure wholesale.

I'd put it differently: it's paying up hundreds of billions in infrastructure to have some sort of gain.

And which gain is that exactly?

I see a lot of "the world is dumb but I am smart" comments in this thread but I saw no one presenting any clear advantage or performance improvement claim about hypothetical replacements. I see a lot of "we need to rewrite things" comments but not a single case being made with a clear tradeoff being presented. Every single criticism of TCP/IP in this thread sounds like change for the sake of change, and changes that aren't even presented with tangible improvements in mind or a clear performance gain.

Wouldn't that explain why TCP is so prevalent, and no one in their right mind thinks of replacing it?

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by fragmede on Monday October 31, 2022 @04:21AM (#33403223)
I mean the goal is more performance, especially if you can get more performance out of the same hardware. Faster setup times, faster connections, more connections, maybe faster teardown. Lower contention on saturated links. Inside of the datacenter is a controlled environment where something like that could work. Replacing TCP over the Internet at large is going to be an uphill battle. Still, if we're replacing the whole thing, then simpler code on the client and server end would be nice.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by simplotek on Monday October 31, 2022 @10:13AM (#33405958)
> I mean the goal is more performance, especially if you can get more performance out of the same hardware.

Are there actual numbers demonstrating this?

I mean, people are advocating wasting billions revamping infrastructure. What kind of performance are you hoping to buy with those billions? And are those gains worth it, or is just sake for the sake of change?

Sometimes things are indeed good enough.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by iso1631 on Monday October 31, 2022 @06:21AM (#33403900)
> Every single criticism of TCP/IP in this thread sounds like change for the sake of change, and changes that aren't even presented with tangible improvements in mind or a clear performance gain.

It amuses me that many of those saying "we need to change" are the same ones that bemoan it when car manufacturers remove buttons or make glove boxes operational from a touch screen because they can.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by wmf on Monday October 31, 2022 @12:39PM (#33408364)
It's figure 1 in the paper. Homa is over 10x faster than TCP (presumably CUBIC).
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by dec0dedab0de on Monday October 31, 2022 @02:41AM (#33402683)
We should have a way to forward every single layer of the stack across each hop, and return back each layer of the stack, so that we can programmatically determine the exact causes of network issues

The only way this would be safe is if you own the entire network including the computers, otherwise you're leaking info.

That said, I used to do this with Wireshark apliances when I was a network engineer, which is cheating but I could grab all the packets from a transaction and export to the Wireshark desktop app to analyze.

I also remember the Cisco reps trying to sell something that would do something closer to what you describe, but it that was almost a decade ago and a proprietary protocol anyway.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by simplotek on Monday October 31, 2022 @03:33AM (#33402942)
> Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo.

If this was true then how do you explain that the likes of AWS, the same company who ended up investing in developing their own processor line, doesn't seem to agree that none of the pet peeves you mentioned are worth fixing?

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by emn13 on Monday October 31, 2022 @05:42AM (#33403653)
It's not obvious to me that replacing TCP really is harder than designing your "own" chip. Scarequotes here because those graviton chips (that's what you're referring to, I think?) are of course ARM chips, so they're not designing something fresh; they're adapting a very mature design to their own needs. In terms of interoperability, a custom chip based on a standard design is probably a simpler, more locally addressable problem than new network protocols.

Isn't it plausible that graviton was designed yet TCP retained simply because graviton as a project is easier to complete successfully?

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by xxpor on Monday October 31, 2022 @10:34AM (#33406302)
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by robertlagrant on Monday October 31, 2022 @03:59AM (#33403115)
> There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo.

This is just not true. Stuff needs to be funded and worth doing, and the internet, like almost everything, is built on making things worth paying for, but there are also loads of improvements everywhere are being made.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by friendzis on Monday October 31, 2022 @04:29AM (#33403267)
> It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP as some sort of universal tunnel encapsulation for all application-layer protocols

I think this is more of an artefact of horizontal scaling and port-contention. De-facto standard discovery mechanism DNS does not work with ports, so "well-known port" abstraction kinda fails. Http as tunnel mostly avoids/sidesteps this problem.

> We should have a way to forward every single layer of the stack across each hop, and return back each layer of the stack, so that we can programmatically determine the exact causes of network issues, automatically diagnose them, and inform the user how to solve them.

This is weird take or I don't understand it. If you can communicate with an edge node in another network, but the edge node has issues communicating with some inner node (on your behalf), then, as a user, you have no hope of fixing that connectivity issue anyway, regardless of whether layered approach is used or not. This may be related to previous point about http as universal tunnel. Yes, this is a problem, but in a way that communications are effectively terminated at the edge node and monstrosity of stuff happens behind the scenes

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by patrec on Monday October 31, 2022 @04:45AM (#33403345)
> De-facto standard discovery mechanism DNS does not work with ports

Yes, it does, see SRV records.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by friendzis on Monday October 31, 2022 @05:34AM (#33403606)
I meant DNS A/AAAA queries with preconfigured/well-known ports being the default. While some applications/protocols/services do use some port discovery mechanism, I would argue it is nowhere close to being de-facto standard.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by bewo001 on Monday October 31, 2022 @10:12AM (#33405938)
So true, but how many developers know about them? The API situation does not help either.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by AtNightWeCode on Monday October 31, 2022 @12:43PM (#33408437)
I would say ports are mainly a problem on layers below transport even though some tech overuse ports.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by kortilla on Monday October 31, 2022 @04:50AM (#33403366)
> There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo.

This is severe bullshit on two fronts:

- there is an immediate return on value - Google was driving this a decade+ ago for improvements in the data center (things like doubled+cancelable rpc, tcp cubic, quic, etc)

- academia constantly attempts to make these improvements as well because researchers are super incentivized to dethrone tcp for the glory. There are constant attempts to re-invent various layers (IP, tcp, the non existent upper layers of the OSI, etc) that come out of academic conferences every year.

The reason we’re still here is because our current stacks have been heavily optimized and tooled for production workloads. NICs can transparently re-assemble TCP segments for the OS and they can segment before transmit. You have to have a damn good value prop to throw away everything from software and hardware to careers and curriculum. It has to be a shitload better than the security nightmare of “return back each layer of the stack”.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by gjulianm on Monday October 31, 2022 @04:58AM (#33403405)
> We should have a way to forward every single layer of the stack across each hop, and return back each layer of the stack, so that we can programmatically determine the exact causes of network issues, automatically diagnose them, and inform the user how to solve them. But right now, you need a human being to jump onto the user's computer and fire up an assortment of random tools in mystical combinations and use human intuition to divine what's going on, like a god damn Networking Gandalf. And we've been doing it this way for 40+ years.

I work in a company that builds network troubleshooting/observability tools and we have some pretty experienced analysts to tell you what's wrong with the network. With that context, your idea of having any tool automatically diagnosing network issues is a pipe dream.

The problem with networks is that they're very complex systems, with multiple elements along the way, made by different manufacturers, often with different owners, failures aren't always easily reproducible, and with human configuration (and therefore errors) almost every step of the way. Even if a tool that "returns each layer of the stack" would be useful, it still would be far from enough to diagnose issues.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by EricE on Monday October 31, 2022 @10:47AM (#33406502)
"The problem with networks is that they're very complex systems, with multiple elements along the way, made by different manufacturers, often with different owners" Ah, how people forget the early days of networking. I remember vividly the early days of the Networld/Interop trade show - Interop was in the name because if, as a vendor, your equipment couldn't integrate with the show network they would throw your booth off the show floor.

That's how bad interoperability in the early days was!

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by cdogl on Monday October 31, 2022 @05:00AM (#33403416)
I’ll defer to experts on the network-layer problems but im not sure what you see as the problem with converging on HTTP. It’s awkward and inelegant, but as an a backend application developer I never feel like it gets in my way.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by jstimpfle on Monday October 31, 2022 @05:26AM (#33403564)
> It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP

If you have ever used multiple TCP or UDP connections in parallel on a single machine (doesn't matter if server or client) then you should realize that ports are required.

Apart from that, you can run HTTP on other ports than 80. You can also use HTTP to load balance or do service discovery by means of redirects. (Caveat, I don't work in this field and can't say how solid the approach works in practice).

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by twawaaay on Monday October 31, 2022 @06:08AM (#33403800)
I don't think you realise why this is so hard.

The basic reason is that software at every level expects TCP/IP. And you can't drop in a translation layer because it will require at least the same amount of overhead as "real" TCP/IP.

It is not a local problem, it is a global problem that affects basically every single piece of non-trivial software in existence.

Even if you construct your datacenter with the new protocol you will run into problems that you can't run anything in it. Want Python? Sorry, have to rewrite it. And every Python library. And every Python application. Then you need to deal with problems that people who can run their scripts on their machines can't run them in datacenter. And so on.

The reason nobody wants to do this is that they would be investing huge amount of money to solve a problem for everybody else. Because the only way to make TCP/IP replacement work is to make it completely free and available to everybody.

There are much better ways to allocate your funds and precious top level engineers that let them distance themselves from competition temporarily.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by starfallg on Monday October 31, 2022 @06:15AM (#33403847)
> We still lack any way to communicate up and down the stack of an entire transaction, for example for debugging purposes. We should have a way to forward every single layer of the stack across each hop, and return back each layer of the stack, so that we can programmatically determine the exact causes of network issues, automatically diagnose them, and inform the user how to solve them. But right now, you need a human being to jump onto the user's computer and fire up an assortment of random tools in mystical combinations and use human intuition to divine what's going on, like a god damn Networking Gandalf. And we've been doing it this way for 40+ years.

This violates the principle of encapsulation that the entire field of networking is based on, not to mention an massive security hole.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by KaiserPro on Monday October 31, 2022 @06:18AM (#33403879)
> We should have a way to forward every single layer of the stack across each hop, and return back each layer of the stack, so that we can programmatically determine the exact causes of network issues

Thats virtual networking. but that introduces latency if its not well configured.

> But right now, you need a human being to jump onto the user's computer and fire up an assortment of random tools in mystical combinations and use human intuition to divine what's going on, like a god damn Networking Gandalf

not really, assuming you have the right fabric, its nowhere near as hard as that. Plus you seem to be forgetting that there is more to the network than TCP. There is a whole physical layer that has lots of semantics that greatly affect how easy it is to debug higher levels.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by arka2147483647 on Monday October 31, 2022 @07:05AM (#33404188)
> It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP as some sort of universal tunnel encapsulation for all application-layer protocols. And then there's all the non-backend problems!

The paper argues the in '3.1 Stream orientation' section, that stream orientation is a problem for TCP, and says that most apps send messages instead, and the better protocol should handle messages, natively, etc. Which is a good point I think.

But back to TCP. What do you do, if you need to send Messages between applications in TCP? Preferably those Messages would be encrypted also.

You could make up your own protocol, but you probably would rather not! So you use something that is readily available, and does messages, encryption, etc. Would be nice if there were also a ready to use load balancers, caches, tools to debug it, etc

Now, what would be such a protocol.

Why HTTPS, of course.

So I kind of think that the lack of a low level Message Protocol has lead us, as an industry, to coalesce these features bit-by-bit on top of HTTP. It's not perfect by any means, but it does the job.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by pclmulqdq on Monday October 31, 2022 @08:03AM (#33404607)
HTTPS adds a tremendous amount of overhead to give you messaging. It's a lot better from a hyperscaler's perspective to replace TCP and not use the byte stream abstraction. After all, networks send messages. It's silly to throw that away at one layer and try to get it back at the next layer.
Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by guenthert on Monday October 31, 2022 @12:49PM (#33408556)
> It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP as some sort of universal tunnel encapsulation for all application-layer protocols.

Nobody forces them though. It would be much easier to publish a standard port number mapping than to develop a (or multiple) new protocols. Now you just need to motivate people to use it.

Re:Ousterhout: It's time to replace TCP in the Datacenter [pdf] by still_grokking on Monday October 31, 2022 @09:01PM (#33415668)
> There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo.

I would affirm that. It's imho true for almost everything in IT tech.

How computers "work" today is just pure madness when looked anyhow closer.

Everything's a result of some "historic accidents" back in the days, and from that the usual race to the bottom caused by market powers.

Nobody is willing to touch any of the lower layers no matter how crazy they are form today's viewpoint. We just shovel new layers on top to paper over the mistakes of the past. Nothing gets repaired, or actually what would be more more important, rethought form the ground up in light of new technological possibilities and changed requirements.

I understand from the economic standpoint how this comes. But I'm also quite sure we didn't make any fundamental improvements in the last 50 years of computing.

That's a very bad sign when everything in a field that's not even really 100 years old is frozen in time since 50 years because everything's so fragile and complex that fundamental changes aren't possible. This looks like a text book example of a house of cards…

Given how vital IT tech is to modern life I fear that this will crash at some point in the worst way possible.

And even if it won't crash, which is really strongly hope, we will never have nice things again as nothing of the old rotten things can be reasonably changed.