News for hackers. Stuff that matters.
Re:Lying to TCP makes it a best-effort streaming protocol by makira on Friday February 02, 2018 @01:22PM (#16292815)
So, UDP.
Re:Lying to TCP makes it a best-effort streaming protocol by mordero on Friday February 02, 2018 @01:28PM (#16292861)
I don't think that is quite true. I believe with UDP there is no promise of packets being received in order. I think this article is saying that you still get the benefits of processing the packets in the order received, but you don't have to worry about the latency of waiting for the re-transmission of any packets.
Re:Lying to TCP makes it a best-effort streaming protocol by vvanders on Friday February 02, 2018 @01:36PM (#16292948)
It's so trivial to add ordering to UDP, it's really the right protocol to use here.

Subverting TCP leads to all sorts of problems around congestion(which you can no longer filter on because which TCP streams are being non-compliant?) that it just should not be done.

Re:Lying to TCP makes it a best-effort streaming protocol by JoeAltmaier on Friday February 02, 2018 @05:46PM (#16294848)
TCP layer also gives you pacing, congestion control and flow control. Which UDP, you'd have to do yourself?
Re:Lying to TCP makes it a best-effort streaming protocol by JoeAltmaier on Friday February 02, 2018 @05:59PM (#16294943)
And its not totally trivial to reorder UDP. Say you receive packets 1,2,3,5. How long to you wait for '4'? Maybe it was dropped; maybe its coming.

Then you get 6,7. Is 4 still out there? You've got 3 packets in your pocket, waiting for 4. That adds up too.

TCP gives you some idea of what packets you SHOULD have received, so you can respond better. UDP doesn't have any windowing etc so you have no idea.

Re:Lying to TCP makes it a best-effort streaming protocol by vvanders on Friday February 02, 2018 @06:12PM (#16295016)
I guess I don't really see the big issue with that. It's not like windowing and congestion control is some kind of black magic. It's spelled out pretty cleanly in the TCP RFC and pretty straight-forward to reimplement.

Generally if you're hitting cases where TCP is causing you grief and you need to reach for UDP you've already got enough context to understand your congestion problems/etc.

We've been doing this in game-dev for decades, ditto the voip space so it's not like you don't have a wealth of knowledge to draw from if you're really stumped.

Re:Lying to TCP makes it a best-effort streaming protocol by JoeAltmaier on Friday February 02, 2018 @07:51PM (#16295448)
If you just use TCP again you haven't done anything. The whole point is to avoid latency.

Most folks use some UDP-based protocol package instead of reinventing the wheel. Its not rocket science, but it isn't trivial. Defining your own packets to do all the flow stuff is just work, like any other programming task.

Re:Lying to TCP makes it a best-effort streaming protocol by vvanders on Friday February 02, 2018 @08:34PM (#16295613)
I don't think I was suggesting using TCP, I was suggesting implementing the features you like from TCP into your stack if you really need them. You can do congestion control without retry, etc.

I've built variations of UDP based protocols 4 or 5 different times over my career. I'm literally in the middle of this right now with the radio framing protocol I've been developing. I really think you're making it out to be much harder than it is.

Re:Lying to TCP makes it a best-effort streaming protocol by tonyg on Saturday February 03, 2018 @04:52AM (#16296955)
I was delighted when DCCP appeared: https://en.wikipedia.org/wiki/Datagram_Congestion_Control_Pr...

It focuses narrowly on a congestion control protocol, and is intended to be combined which whichever datagram-based protocol you have lying around that might be suffering from congestion issues.

Re:Lying to TCP makes it a best-effort streaming protocol by ramenmeal on Friday February 02, 2018 @06:39PM (#16295136)
Isn't the pacing and congestion control based on ACK's though? And this is suggesting to ACK everything. I feel like I'm missing something.
Re:Lying to TCP makes it a best-effort streaming protocol by marcosdumay on Friday February 02, 2018 @08:32PM (#16295602)
> It's so trivial to add ordering to UDP

Specially when compared with creating and using a user space IP protocol like was done here, or adding a new one into the kernel.

Re:Lying to TCP makes it a best-effort streaming protocol by ramzyo on Friday February 02, 2018 @05:40PM (#16294796)
I'm not sure I understand the distinction. How could TCP guarantee that packets will be received in order without re-transmissions? Re-transmission is a mechanism for making this a guarantee. If the receiver just ACKs everything it gets, then isn't that effectively making no guarantees about the order?
Re:Lying to TCP makes it a best-effort streaming protocol by JoeAltmaier on Friday February 02, 2018 @05:45PM (#16294841)
TCP layer of your IP stack does re-ordering and presents them to the client in order. UDP layer doesn't. So by acking every packet, TCP layer will still present what it DOES receive in order.
Re:Lying to TCP makes it a best-effort streaming protocol by ramzyo on Friday February 02, 2018 @08:52PM (#16295676)
Right, but UDP also presents what it receives in order - so what's the advantage of forcing TCP to behave this way? I struggle to think of a practical use case where either a) UDP or another protocol wouldn't be selected [e.g. in a VoIP system] in the design phase b) using TCP in this non-standards compliant way would be nothing more than a short-term bandaid because of other constraints (e.g. can't change Layer 4 to something non-TCP).
Re:Lying to TCP makes it a best-effort streaming protocol by JoeAltmaier on Saturday February 03, 2018 @08:35AM (#16297552)
Absolutely not. UDP delivers packets as received.

Don't underestimate how often packets are received out of order. There's even a consumer DSL modem that swaps every odd UDP packet with every even one - I had to compensate for this in a VOIP product. Using TCP in this bastardized way would cure that. That said, I tend to agree its a poor idea to use TCP in this way. The famous book on IP used to list 8 protocol possibilities (only 2 commonly survive today, UDP and TCP) of which streaming and packet reordering was a valid combination (without acking/retransmitting). Don't know what it was called, but that's what being attempted here.

Re:Lying to TCP makes it a best-effort streaming protocol by ramzyo on Saturday February 03, 2018 @12:33PM (#16298520)
I think we’re operating on different definitions of in-order and as received. TCP delivers packets in order, but perhaps not as received, if it had to request retransmission of a dropped packet. UDP delivers packets in order that they were received. Doing what the article suggests would make TCP also deliver packets in the order that they were received. No?
Re:Lying to TCP makes it a best-effort streaming protocol by JoeAltmaier on Saturday February 03, 2018 @12:44PM (#16298574)
TCP won't nack a packet when one is perceived as 'missed' (but really out-of-order); that's clear from the premise (ack everything).

But reordering also happens simply by examining received packets in a burst, and putting them right.

So its six of one and a half-dozen of the other I guess. Sorry for coming off so abrupt.

Re:Lying to TCP makes it a best-effort streaming protocol by ramzyo on Saturday February 03, 2018 @01:34PM (#16298818)
No problem, thanks for clarifying! So basically the benefit to the TCP NACK approach is that the TCP layer will also do a sort on the packets received?
Re:Lying to TCP makes it a best-effort streaming protocol by JoeAltmaier on Sunday February 04, 2018 @10:27AM (#16303077)
Yeah, which is imperfect but there you are. It means sometimes out-of-order packets will be reordered, and sometimes dropped (since TCP acked it, the (existing) TCP code will discard the out-of-order packet as 'duplicate'. Which turns out works pretty well in practice, since out-of-order are almost always in a burst (no delay inter-packet)
Re:Lying to TCP makes it a best-effort streaming protocol by andreasvc on Friday February 02, 2018 @01:29PM (#16292878)
No. UDP is connectionless, which makes it harder to use with NAT.
Re:Lying to TCP makes it a best-effort streaming protocol by andreasvc on Saturday February 03, 2018 @12:18PM (#16298463)
To clarify, what I meant is that with TCP, I can set up a two-way communication channel, even if I'm behind a NAT/firewall I don't control. As far as I understand, with UDP this is harder (i.e., does not work with all NAT types), because UDP does not establish a connection, and it does not provide a two-way communication channel. However, I am not up to date on NAT traversal techniques, so I might be wrong.
Re:Lying to TCP makes it a best-effort streaming protocol by gsich on Friday February 02, 2018 @06:15PM (#16295028)
Which was maybe a problem 20 years ago.

But probably not.

Re:Lying to TCP makes it a best-effort streaming protocol by xxpor on Friday February 02, 2018 @01:33PM (#16292913)
Not sure why you say that, UDP conntracking has been around for a long time.
Re:Lying to TCP makes it a best-effort streaming protocol by trav4225 on Friday February 02, 2018 @01:44PM (#16293044)
I suspect that what andreasvc meant is that the default "NAT" configuration of most consumer-grade gear is such that it will block UDP (unless some other mechanism such as UPnP is used)...
Re:Lying to TCP makes it a best-effort streaming protocol by lmns on Friday February 02, 2018 @01:54PM (#16293153)
That's true for TCP as well. Almost all consumer-grade routers block all incoming connection attempts, not just UDP.

From a stateful firewall's point of view both UDP and TCP have state.

Re:Lying to TCP makes it a best-effort streaming protocol by trav4225 on Friday February 02, 2018 @01:58PM (#16293207)
Right. I was assuming that the context here was connections being initiated by the client (as most are).
Re:Lying to TCP makes it a best-effort streaming protocol by kelnos on Friday February 02, 2018 @02:46PM (#16293630)
... in which case blocking is not an issue. Consumer-grade NAT hardware will no more block client-initiated UDP than it'll block client-initiated TCP, at least not without extra configuration.
Re:Lying to TCP makes it a best-effort streaming protocol by noselasd on Friday February 02, 2018 @01:51PM (#16293120)
But it doesn't. Consumer gear NATs UDP pretty much as nice as it does TCP.
Re:Lying to TCP makes it a best-effort streaming protocol by trav4225 on Friday February 02, 2018 @02:03PM (#16293261)
Hmm, perhaps you are right... I switched from consumer gear to more "enterprise-y" gear a few years back, so my data points may be a bit outdated.
Re:Lying to TCP makes it a best-effort streaming protocol by testvox on Friday February 02, 2018 @04:55PM (#16294517)
I've been using normal consumer routers for NATed home connects since 2004 or so and I've never had an issue with outgoing UDP. It's required for basically every video game after all.
Re:Lying to TCP makes it a best-effort streaming protocol by da_chicken on Friday February 02, 2018 @02:17PM (#16293380)
If NAT blocked UDP, the DNS client on home computers would never work.
Re:Lying to TCP makes it a best-effort streaming protocol by icebraining on Friday February 02, 2018 @02:23PM (#16293418)
I think most routers set themselves as the DNS server, so NAT is not in effect (the computer only sends the request to a local address) unless you define a custom DNS server, which isn't common for home users.

That said, I've never seen a router that didn't allow UDP packets to flow back to the origin client.

Re:Lying to TCP makes it a best-effort streaming protocol by da_chicken on Saturday February 03, 2018 @01:15PM (#16298711)
> I think most routers set themselves as the DNS server

DNS forwarders like dnsmasq are a relatively recent inclusion in home routers. Sure, they've been there for 10 years or so, but they weren't there for the 5+ years before that. Before Linux took over the embedded OS on home routers, the DHCP servers just passed the DNS configuration that the WAN port got from the ISP, and you can still do that now if you want. That's why nslookup.exe and dig still work on your workstation when you specify an external DNS server instead of the one your DHCP server on your home router gives you.

> That said, I've never seen a router that didn't allow UDP packets to flow back to the origin client.

Which is the point I was making.

Re:Lying to TCP makes it a best-effort streaming protocol by johncolanduoni on Friday February 02, 2018 @04:44PM (#16294439)
Most consumer-grade gear can easily handle two-sided NAT transversal with merely a STUN server (no UPnP or relaying required).
Re:Lying to TCP makes it a best-effort streaming protocol by mnd999 on Friday February 02, 2018 @01:40PM (#16293007)
Then don’t use NAT, it’s a hack anyway. IPv6 is not exactly a new idea.
Re:Lying to TCP makes it a best-effort streaming protocol by noselasd on Friday February 02, 2018 @01:48PM (#16293086)
Why does it matter that IPv6 is old, if your users are only using IPv4 ?
Re:Lying to TCP makes it a best-effort streaming protocol by funkaster on Friday February 02, 2018 @01:26PM (#16292844)
not exactly:

> If it’s used that way, it makes TCP a reliable, in-order protocol.

the key here is "in-order".

Re:Lying to TCP makes it a best-effort streaming protocol by Dylan16807 on Friday February 02, 2018 @01:31PM (#16292896)
It's absolutely trivial to put a sequence number in UDP packets.

The question is whether you want the other mechanisms of TCP: handshaking, (limited) retransmission, flow control...

Re:Lying to TCP makes it a best-effort streaming protocol by digi_owl on Friday February 02, 2018 @01:39PM (#16292983)
Never mind that most sensible streaming formats will include data in packets that allows the receiver to rebuild missing packets (up to a threshold).
Re:Lying to TCP makes it a best-effort streaming protocol by isostatic on Saturday February 03, 2018 @07:56AM (#16297430)
A typical rtp stream will have smpte fec on top, allowing burst lost of say 20 packets, or random loss theoretically upto 5%

In the last 10 years of streaming rtp over the Internet the vast majority of failures is bursty. Reordering is more rare than you'd expect.

Looking at one sample from India to Europe over a period of 6 weeks, my 30mbit rtp stream was fine 99.998% with fec. The rest of the time that's why I dual stream, either timeshift (send the same packet again 100ms later - as I said most outages are time based), or dual path (although if the packets traverse the same router en route there are issues), or even both.