Newline-delimited != SOTA
https://en.wikipedia.org/wiki/Apache_Arrow :
> Arrow allows for zero-copy reads and fast data access and interchange without serialization overhead between these languages and systems
JSON lines formatted messages can be UTF-8 JSON-LD: https://jsonlines.org/
Linux networking is now all built on eBPF.
> In general when we write to a socket, or a file, or in our case a pipe, we’re first writing to a buffer somewhere in the kernel, and then let the kernel do its work. In the case of pipes, the pipe is a series of buffers in the kernel. All this copying is undesirable if we’re in the business of performance.
> Luckily, Linux includes system calls to speed things up when we want to move data to and from pipes, without copying. Specifically:
> - splice moves data from a pipe to a file descriptor, and vice-versa.
> - vmsplice moves data from user memory into a pipe.
> Crucially, both operations work without copying anything.
But then to scale to more than one node, everything has to be copied from the pipe buffer to the network socket buffer; unless splice()'ing from a pipe to a socket is Zero-copy like sendfile() (which doesn't work with pipes IIRC).
"SOCKMAP - TCP splicing of the future" (2019) https://blog.cloudflare.com/sockmap-tcp-splicing-of-the-futu...
It's all a big pile of stateful daemons notifying other daemons with a billion race conditions and zero debugging capabilities, like a parody of how not to create reliable systems. When you have shell scripts parsing JSON messages, you know it's over.