News for hackers. Stuff that matters.
Re:Ubus (OpenWrt micro bus architecture) by westurner on Sunday August 27, 2023 @03:45PM (#37285957)
But then how to add datatypes - schema - to pipes? How do we add a column to the messages on the pipe?

Newline-delimited != SOTA

https://en.wikipedia.org/wiki/Apache_Arrow :

> Arrow allows for zero-copy reads and fast data access and interchange without serialization overhead between these languages and systems

JSON lines formatted messages can be UTF-8 JSON-LD: https://jsonlines.org/

Linux networking is now all built on eBPF.

Re:Ubus (OpenWrt micro bus architecture) by packetlost on Sunday August 27, 2023 @03:59PM (#37286081)
The same way you add it to any other SOCK_STREAM? How do we add types and schemas to TCP connections?
Re:Ubus (OpenWrt micro bus architecture) by westurner on Sunday August 27, 2023 @04:09PM (#37286185)
But then when I need to scale to more than one node with multiple cores, do pipes scale?
Re:Ubus (OpenWrt micro bus architecture) by packetlost on Sunday August 27, 2023 @05:02PM (#37286607)
Re:Ubus (OpenWrt micro bus architecture) by westurner on Sunday August 27, 2023 @08:54PM (#37288366)
https://mazzo.li/posts/fast-pipes.html#splicing :

> In general when we write to a socket, or a file, or in our case a pipe, we’re first writing to a buffer somewhere in the kernel, and then let the kernel do its work. In the case of pipes, the pipe is a series of buffers in the kernel. All this copying is undesirable if we’re in the business of performance.

> Luckily, Linux includes system calls to speed things up when we want to move data to and from pipes, without copying. Specifically:

> - splice moves data from a pipe to a file descriptor, and vice-versa.

> - vmsplice moves data from user memory into a pipe.

> Crucially, both operations work without copying anything.

But then to scale to more than one node, everything has to be copied from the pipe buffer to the network socket buffer; unless splice()'ing from a pipe to a socket is Zero-copy like sendfile() (which doesn't work with pipes IIRC).

"SOCKMAP - TCP splicing of the future" (2019) https://blog.cloudflare.com/sockmap-tcp-splicing-of-the-futu...

Re:Ubus (OpenWrt micro bus architecture) by packetlost on Sunday August 27, 2023 @11:03PM (#37289221)
sendfile is implemented with splice, but the scenario in which splice is actually zero-copy is not well defined (man pages just say "it's generally avoided"), so you'd have to dig into your kernel source to know for sure. That being said, chances are the cost of copying the data would be far outpaced by the cost of serializing over the network anyways, so you would want to architect your application in a way that you're sending as little data over the network as possible if performance is of maximum concern.
Re:Ubus (OpenWrt micro bus architecture) by westurner on Monday August 28, 2023 @12:00AM (#37289579)
I haven't checked, but by the end of the day, I doubt eBPF is much slower than select() on a pipe()?