News for hackers. Stuff that matters.
Re:What developers should know about TCP by Unklejoe on Friday May 15, 2020 @03:25PM (#23196313)
> stream of bytes

I've always wondered: What's the best/defacto way to delimit this back into packets at the application level on the receiving end?

I would think the obvious approach would be to insert some magic word into the stream so that you can re-sync.

Or is this not an issue since you know that once you're connected, you'll never drop a single byte, therefore, the only way to get out of sync would be a program error?

Re:What developers should know about TCP by jstanley on Friday May 15, 2020 @03:28PM (#23196337)
You will never drop a single byte.

If you need some packet-oriented messaging, you could use something like http://jsonlines.org/ (i.e. JSON messages separated by newline characters), or https://github.com/protocolbuffers/protobuf if it's more performance-critical.

Re:What developers should know about TCP by timeinput on Friday May 15, 2020 @05:21PM (#23197589)
Protobuf isn't self delimiting so you still have to have some extra packet wrapper around it to say the length.

I like zeromq to get to a packet based system.

Re:What developers should know about TCP by mytailorisrich on Friday May 15, 2020 @03:36PM (#23196425)
The standard way is to include explicit information on the length of the message that is following.

For example if the message is x bytes long then you first send 'x' then you send the x bytes of the message.

Or your messages have a defined header that contains the length of the message payload.

Re:What developers should know about TCP by vasilvv on Friday May 15, 2020 @06:52PM (#23198455)
It will never get out-of-sync because TCP guarantees that the bytes will be delivered in the same order they've arrived.

The best approach is typically put a length in front of every message. The good things about that approach are:

1. The receiver can allocate buffer that is exactly the size it needs to fit the message. 2. The receiver can check whether the message is too long before seeing the entire message.

The only disadvantage is that you have to know the length of all messages in advance.

Re:What developers should know about TCP by kccqzy on Friday May 15, 2020 @07:22PM (#23198678)
Definitely be sure to check the length though. Imagine a mistaken client trying to send HTTP, but of course the first four bytes "HTTP" when interpreted as a 32-bit integer, whichever endian, is an absurdly large buffer.
Re:What developers should know about TCP by nucleardog on Friday May 15, 2020 @07:57PM (#23198963)
I mean, at this point you're effectively defining a new network protocol (or, you will be shortly once you implement ways to work around all the other issues you're going to run into). I'd go all-in from the start and start every packet with a magic string/byte sequence of your own, a length, and probably a version code just to make it extensible.

Or see if there's an existing protocol you can abuse for what you want. If it's transactional, you get a pretty big ecosystem of battle-tested clients/servers/proxies/etc if you use HTTP.

Re:What developers should know about TCP by userbinator on Saturday May 16, 2020 @02:24AM (#23200932)
A 16-bit length (64k max message size) is usually sufficient, or even 24-bit (16M max) if you really feel the need, but 32 bits is far more than should be needed for parsing messages in memory; it would be fine for a streaming application, however (in which case a 64-bit length wouldn't be a bad idea either.)
Re:What developers should know about TCP by kccqzy on Saturday May 16, 2020 @03:48AM (#23201313)
Good advice. I was actually referring to this: https://rachelbythebay.com/w/2016/02/21/malloc/ I read this article a long time ago, and yet I made a similar mistake in my own code.
Re:What developers should know about TCP by genpfault on Friday May 15, 2020 @03:51PM (#23196628)
Re:What developers should know about TCP by patrickmcmanus on Friday May 15, 2020 @08:14PM (#23199069)
preamble of chunk length and 1 bit for end-of-message indicator.. if you only do chunk length you will eventually find you can't stream but want to.

or just use http.