This kind of issue happens all the time with structs that a kernel, driver etc. expose to userspace: https://j00ru.vexillium.org/papers/2018/bochspwn_reloaded.pd...

Not sending the structure over the network is insufficient, and rigorous memset is a clear solution to this problem.

Also, C structure layout for a specific ABI is strictly determined; if that weren’t the case you couldn’t compile two C compilation units with different flags/compilers and expect them to work.

Also don't expect memset() to actually be a library call at all.

And most importantly dont expect memset to be executed at all. Most -O2 compilers happily optimize the memset away if if doesn't care about the sideeffects. Therefore there exists the "secure" memset variants, which are the insecure variants adding a simple compiler barrier, so it isn't optimized away.

And there exists the very few really secure memset_s variants with a real memory barrier which guarantees memset to be executed in load and store order, flushing the cache, so that Spectre attacks are mitigated. security-relevant crypto libraries, like openssl and variants don't care about that, because memory barriers are "too slow". only the Linux kernel does it properly. It's still a mess.

you can write your own memset that bypasses all of that. Its just 2 lines of code.

And that will be slower than calling memset() unless you have a really smart compiler.

I don't trust any memset at all. Literally every single memset I looked at was either broken, too slow or insecure. Mostly the wellknown glibc, freebsd, msvcrt and compiler implementations.

That's quite a bold claim. Care to explain further?

I already explained it. Noone implements memory barriers in its safe variants, only compiler barriers. And the trivial variants work byte-wise, not word-wise, which is usually 8x slower.

> Also, C structure layout for a specific ABI is strictly determined

That's true, but when sending stuff over the network you don't know if the receiver uses the same ABI.

Changing compiler or flags won't change structs, unless you do something crazy. If it did, you couldn't link C libraries compiled with one compiler with programs compiled on another.

You can't be sure you can send such structures to different OSes or CPUs however.

> If it did, you couldn't link C libraries compiled with one compiler with programs compiled on another.

You cannot do this unless the compilers have the same ABI, which is not part of the standard.

It depends upon the architecture and compiler. One C compiler for the MC68000 might have int as 16 bits (since externally, the MC68000 had a 16-bit external bus) but another one have ints as 32 bits (since the MC68000 can handle 32-bits internally). This was more of an issue during the 80s and 90s when you had more commercial C compiler available than today.

That is what stdint is for.

> Sending structures that include padding or that you don’t know the exact layout of over the network is another thing you shouldn’t do, period.

What if you do know the exact layout and it happens to have a gap between fields?

Tbh I'm not a network person, but my naive brain says instead of casting to char* and writing to the buffer, couldn't you just write out individual fields to a buffer so you control every byte? That seems smarter and safer.

It depends. It was common case for years in games to just cast your networked data to an array and send over the network. Nowadays it's much more in fashion to pick a serialization library to sit in between, as PCs are faster. But one might choose not to depending on the problem space. Embedded devices might not have the spare cycles or the space for a full serialization library.

So...you're not wrong, but it's basically the same reason people use C over something with more ergonomics.

Edit: the last time I wrote a networking library, I also just cast to raw structures, mainly because it was less work (didn't want to integrate a serialization library on both sides of the link). This was fine for a while (and very fast), but also hid some bugs in the protocol, and made sending arrays of data very ugly. Eventually we put protobuf on top, and things ended up much nicer.

How does sending raw memory to the socket work if you have a struct filled with pointers?

I think they simply don't do that with structs that contain pointers?

Indeed, this. If you have data with pointers, you either need to flatten (read: copy) your structure into something appropriate for structuring, or just get a proper serialization layer.

I guess there could be some fun workarounds with ptrdiff_t instead of pointers. "My child is 1000 bytes that way in memory" etc.

That's a pretty classical file serialization trick to pack a bunch of heterogeneous data in one file. A header/manifest starting the file, with an array of names, offsets, and sizes.

Actually works somewhat well for mmap'ed access, too, by using a macro or function to "dereference" "pointers" (offsets).

Or even an OffsetPtr class with overloaded arrow and dereference operator, if you swing that way. :)

What’s the runtime performance of that in-practice? Does x86, x64 and ARM have support for relative offset pointers without any performance impact? And do compilers use them?

Yes, address generation can at the very least do [reg + constant], and on x86, much much more. These are bread and butter optimizations constantly used by compilers all over the place, because this is how field access works.

Do you write a separate serialization function for every single struct and sub-struct you send over a network? And then continue maintaining every single one to keep it consistent whenever a field is added/removed/modified in any struct?

(Though I should mention that macros actually do make this doable, but they're hardly pretty, and I have yet to see many people embrace this approach, if they realize it's possible at all.)

> Do you write a separate serialization function for every single struct and sub-struct you send over a network? And then continue maintaining every single one to keep it consistent whenever a field is added/removed/modified in any struct?

Yes? Of course you do that? Or pick one of a hundred libraries that do that for you - protobuf, cap'n proto, flatbuffers, etc...

That's the protobuf approach, but things like Capnp are different, right? Isn't its wire format its in-memory format? I thought it was closer to the "straight memcpy" approach.

Protobuf serialization functions can turn into a lot of generated code too (read: big binaries) and stomp all over your icache, because they're different for each type. More "data-driven" approaches using type-generic functions with some runtime type-specific data (like proto descriptors&reflection) can be a good tradeoff for code that isn't as "hot". Typically more branchy than a memcpy or a type-specific function though.

There are dozens, if not hundreds of serialization libraries. None of them are going to compare to something like sending a raw struct on the wire, if you need absolute performance. Probably the biggest advantage of things like grpc, flatbuffers, capnproto, etc, are they provide serialization libraries for many different languages. Sending a raw struct will be up to the implementer to get the other side correct.

And by sticking with the serialization libraries with higher performance, like flatbuffers and capnproto, you also lose a little bit of future flexibility, since its typically harder to change the format of these structures compared to higher level libraries.

Author of Cap'n Proto here...

> None of them are going to compare to something like sending a raw struct on the wire, if you need absolute performance.

I don't know about that.

For simple flat structs containing integers only (no pointers), Cap'n Proto is nearly identical to raw structs.

For complex structures with pointers, sending raw structs doesn't work.

Admittedly you may suffer unwanted code bloat linking in the Cap'n Proto library if all you really want to do is send a flat, raw struct.

> And by sticking with the serialization libraries with higher performance, like flatbuffers and capnproto, you also lose a little bit of future flexibility, since its typically harder to change the format of these structures compared to higher level libraries.

I disagree with this. Cap'n Proto's compatibility story is essentially the same as Protobuf and JSON: You can add new fields and old programs will ignore them.

(I suspect Flatbuffers is also similar but I haven't looked at it in a long time.)

That's what I do these days for simple stuff, or for stuff which needs to talk to nonstandard hardware (eg. talking to a PLC, where I need to know the exact bit layout of the packet in order to unpack it manually on the other side).

If you're doing a large project on supported platforms it's probably worth using a pre-existing serialisation library like protobuffers or something.

There is an attribute that lets you specify alignment per-struct, which means you will know the exact layout.

But it is still potentially problematic if not all of your clients have the same endianess. Memcpy and exact layout still fails you there.

RDMA (Infiniband) does DMA to/from network. Why we should not use RDMA?

Hmm, that concern is not just limited to RDMA; any kind of shared memory communication across a trust boundary is vulnerable to info leaks in the padding of the struct. That does seem like a real issue.

Not sure what's the appropriate mitigation. Having to memset is a drag, and it's easy to forget to do it in some place or another. I'm tempted to say that you should only use packed structs for shared memory communication across trust boundaries.

Well, Infiniband is intended mainly for supercomputers. So security is of no real concern within the Imfiniband network as that is implicitly a trusted environment. Access comtrol to that network is the main security boundary.

Edit: Also, lots of simulation codes I am aware of tend to send flat arrays anyway. So I'd say that your concerns are mostly theoretical in nature.

RDMA, both roce and infiniband have some significant limitations, specifically at higher rates. But they're the best we have right now for direct dma.

Serialization is a non-trivial problem. Especially when dealing with different OS, network stacks, and architectures. protobuf alleviates this quite a bit.