If you have the ability to (minimally) parse what is coming out of your pipe, and you know that your writes are not interleaved, and you know the size of the record you're writing before you write it, you can prefix each record with the size, and use this on the reader side to fetch only upto the record length (read(2) syscall and friends allow you to specify a maximum number of bytes to fetch), for each record. Does this answer your question?

I don't believe what you're describing works. You're suggesting reading first the size of a record (stored itself in a fixed-size field), then reading the rest of it?

But unfortunately that is not atomic. Another process could read into the data between your first and second reads.

Not just pipes - Files, too (actually, all write calls). Assuring your records are always smaller than PIPE_BUF and just using O_APPEND works pretty well.

Writes to files less than PIPE_BUF aren't atomic on parallel file systems. e.g. Panasas, Lustre, GPFS, etc.

Clarification: writes to files less than PIPE_BUFF may be atomic, but they aren't necessarily consistent. This is another important property as you want reads following writes to always have the written values. I think I tripped up over myself since in atomicity you expect the 'all written' and 'nothing written' states, but there's also 'all written but nothing visible' which is a third state which feels like it's part of atomicity, but I guess it's really part of consistency.