News for hackers. Stuff that matters.
Re:Flow-IPC: Open-Source Toolkit for Low-Latency Inter-Proce... by ygoldfeld on Thursday April 11, 2024 @02:51PM (#40005504)
Whoa. I’m the lead developer on this - I got to this post totally by accident: was googling for my own Show HN post about this from a couple days ago - and it took me here without my noticing.

There’s some discussion on it in Show HN, and of course I can answer anything here that people might be interested in too. I’m very proud of it and very grateful Akamai gave the resources to open-source it.

I’d like to have a flashier friendlier site with a slick intro video - haven’t had the time to do that stuff - but the substance and API documentation + Manual are very serious and complete, I hope.

All linked off the blog-post!

Re:Flow-IPC: Open-Source Toolkit for Low-Latency Inter-Proce... by ygoldfeld on Thursday April 11, 2024 @02:55PM (#40005544)
(Akamai owns Linode and uses the blog on Linode.com as a developer-oriented blog. So that’s why the link is to there.)
Re:Flow-IPC: Open-Source Toolkit for Low-Latency Inter-Proce... by signa11 on Friday April 12, 2024 @03:24AM (#40010197)
i have done something exactly identical at my current place of employment, and am always inquisitive to see how others have 'stacked-da-cat'.

we _unfortunately_ gravitated towards protobuf's despite my fervent appeal to go with capn-proto. that has caused a cascade of troubles / missed opportunities for optimizations etc. etc.

Re:Flow-IPC: Open-Source Toolkit for Low-Latency Inter-Proce... by fwsgonzo on Saturday April 13, 2024 @01:39PM (#40024655)
I tried to migrate to capn-proto but it just doesn't build on MinGW so I have no choice but to wait. Like you say, it gets worse the more I wait. But, if the APIs are somewhat sane, they should hopefully also be somewhat similar: Able to switch case on oneofs, movable data structures etc.

I don't like that protobuf has recently started linking with abseil, which despite being a good framework, I can't use it if it doesn't build absolutely everywhere I need it to. So, maybe I'll be forced over to CapnProto one of these days?

Re:Flow-IPC: Open-Source Toolkit for Low-Latency Inter-Proce... by Arelius on Saturday April 13, 2024 @07:35PM (#40027198)
Honestly I like capn-proto. But it's community is a bit light. Even Windows which is "supported" has been a bit fix-it-yourself.

If MinGW isn't already supported, if probably expect to have to take that over yourself.

Re:Flow-IPC: Open-Source Toolkit for Low-Latency Inter-Proce... by sgtnoodle on Saturday April 13, 2024 @06:25PM (#40026720)
I've also developed a strikingly similar low latency real-time IPC message bus for work. It also uses sockets with transparent shared memory optimization. In my case, it's the backbone for an autonomous aircraft's avionics. I made everything agnostic to the message scheme, though, and most of the tooling supports an in-house schema, protobuf, JSON, YAML, etc. There's also clients implemented in C++, Rust, Python and Julia.

What troubles has protobuf caused you?

Re:Flow-IPC: Open-Source Toolkit for Low-Latency Inter-Proce... by signa11 on Monday April 15, 2024 @07:07AM (#40038937)
> What troubles has protobuf caused you?

we are primarily a c++ shop, so all comments are from that vantage point. in no particular order following:

     1. extremely bloated generated code.

     2. wire format is not optimal, specifically for large nested messages which trashes the cache on a already severely constrained h/w.

     3. not easy to perform shared-memory exchange for large messages, for example, if you have 'string' types in your messages, those cannot be allocated on arenas.

     4. <this space is for rent>
Re:Flow-IPC: Open-Source Toolkit for Low-Latency Inter-Proce... by nonane on Saturday April 13, 2024 @12:38PM (#40024248)
Cool stuff!

Does Flow-IPC protect against malformed messages? For example a client sending malformed messages to a server process

Re:Flow-IPC: Open-Source Toolkit for Low-Latency Inter-Proce... by sgtnoodle on Saturday April 13, 2024 @06:32PM (#40026778)
Given that it's shared memory based, it seems like there has to be some degree of trust that the participants are well behaved. What do you mean by a malformed message, though? If you're talking about the payload of the message, that seems like a matter of the message scheme you're using. If you're talking about correctness of the IPC protocol itself, integrity checking is unfortunately at odds with latency.
Re:Flow-IPC: Open-Source Toolkit for Low-Latency Inter-Proce... by ygoldfeld on Sunday April 14, 2024 @02:25AM (#40029098)
Reply co-signed!

That said, I'll add that my Manual (https://flow-ipc.github.io/doc/flow-ipc/versions/main/genera...) gets into the topic of safety (with a few words on how this differs from security; this indeed involves trust). So indeed Flow-IPC does not attempt to provide security for malicious/untrusted conversation-partner -- but it does provide a few enabled-by-default safety mechanisms. The Manual page in question: https://flow-ipc.github.io/doc/flow-ipc/versions/main/genera...

Among other things, the capnp-layer (which, as I noted in a recent reply earlier, is optional to use -- you can and sometimes certainly should "just" go native, and/or combine the two approaches) uses an internally-generated token to ensure some kind of terrible bug/thing/problem/act-of-god isn't mis-routing messages. It's a safety mechanism.

But in terms of, say, incompatible schemas or mis-matching native `struct`s -- what if you used different compilers with slightly different STLs on the 2 sides?! -- it is indeed on the developer to not make that error/not converse with untrusted code. Flow-IPC will guard against misuses to the extent it's reasonable though, IMO.

P.S. [Internal impl note] Oh! And, although at the moment it is the protocol version v1 (for all internally-involved protocols at all layers), I did build-in a protocol-version-checking system from the start, so as to avoid shooting ourselves in the foot, in case Flow-IPC needs to expand some of its internal protocol(s) in later versions. At the very worst, Flow-IPC would refuse to establish a channel/session upon encountering a partner with Flow-IPC version such that their protocols are incompatible. (Again -- academic at the moment, since there is only v1 -- but might be different in the future. At that point a new protocol might be developed to be backward-compatible with earlier-Flow-IPCs and just still work; or worst case throw the aforementioned error if not.)

Re:Flow-IPC: Open-Source Toolkit for Low-Latency Inter-Proce... by dang on Saturday April 13, 2024 @10:14PM (#40028109)
That's great! Show HNs are preferred so I've re-upped your original post and have merged the other thread (https://news.ycombinator.com/item?id=40000104) hither.

But readers will probably want to look at the other article as well: https://www.linode.com/blog/open-source/flow-ipc-introductio....

Re:Flow-IPC: Open-Source Toolkit for Low-Latency Inter-Proce... by ygoldfeld on Sunday April 14, 2024 @01:33AM (#40028899)
Whoa (again), super-cool, appreciate that.
Re:Flow-IPC: Open-Source Toolkit for Low-Latency Inter-Proce... by emmelaich on Sunday April 14, 2024 @05:04AM (#40029795)
Your submission is [dead], dunno why. https://news.ycombinator.com/item?id=39987732
Re:Flow-IPC: Open-Source Toolkit for Low-Latency Inter-Proce... by ygoldfeld on Sunday April 14, 2024 @02:06PM (#40033028)