As GP said:

> if you want reliable data transfer over UDP you have to basically implement almost everything TCP does on top of it.

Even QUIC has an ACK-like signal so you will never avoid it as otherwise you will deal with the two generals' problem (https://en.wikipedia.org/wiki/Two_Generals%27_Problem). The reason QUIC is more efficient than TCP is simply due to the fact that TCP can't evolve further due to middleboxes (called ossification, https://en.wikipedia.org/wiki/Protocol_ossification), not that TCP is inherently bad.

Thanks for the explanation, I didn't know that about QUIC. But QUIC is trying to implement reliable delivery for applications, which makes sense as a general purpose protocol. but if I were doing a file transfer app/protocol these days for example I would send hash+Mac+metadata reliably and then check chunks of the file (unreliable+encrypted UDP)as they are decrypted (have to anyways for integrity checking as part if decryption), and peform reliable retransmission request when decryptiom fails. That's what I meant by application layer, application and use-case specific protocols. but perhaps general purpose protocols are still better (I am not sure, I was asking in my post, sorry for not being clear)

You could build this as an application mapping over QUIC. Within a single QUIC connection use reliable streams for the metadata delivery, and unreliable datagrams [1] for the payload. The benefit of a single connection is shared security context and shared fates of different datatype transfers.

[1] - https://www.rfc-editor.org/rfc/rfc9221.html

Then you would generally try to look at probabilistic reliability. One way systems like digital broadcasting requires probabilistic reliability so a lot of research was done here. Digital broadcasting has produced a lot of answers in this area, as you need to deal with both data corruption and missing data. On the data corruption side, you would want to implement a forward-error correction (FEC, https://en.wikipedia.org/wiki/Error_correction_code#Forward_...) to allow recovery of corrupted packets, while on missing data it can be mitigated by fountain codes (https://en.wikipedia.org/wiki/Fountain_code) that is basically FEC but deals with missing pieces instead of incorrect data. This of course does not guarantee 100% delivery but in practical terms this is humanly reliable.

Thank you, that certainly gives me a lot to think about in this area.

With scaling tcp can use window up to 1GB. Which is enough for 1s rtt which is ridiculously high @ 8gbps which is also ridiculous for long haul on a single flow. In practice you probably almost always just parallelize into multiple flows

I had no idea TCP can reach 8gbps over 1s rtt. I suppose folks that tune it and use the right algorithm can get that or multi-threaded transfers working. The problem is most services and applications are not specifically tuned for this.

They don't need to; it is the kernel that implements TCP, and on Linux at least you can easily change the congestion control algorithm that is used and some other parameters. Again, UDP doesn't magically solve anything, and QUIC implements congestion control in userspace, so then you have to wonder: is my QUIC implementation tuned for 8gbps over 1s rtt?

Note that your example of file transfer is a special case that an application can be specifically tuned for; it can have a receive window that is as large as the file it is going to receive, because it can use the file on disk as the backing store for that window, and it can implement an unlimited form of selective ACK. This is really not something that you can do with a generic transport protocol that implements the POSIX sockets API.