But that’s my question, if you choose a binary format, what are the issues to look out for?
Example, I work with Java, Rust, Go, and Python at work. I have specifically had issues with Avro (not my choice) which is really well supported in Java, but not much else.
Binary data support is pretty nice too for avoiding multipart request bodies.
That said, there was a comment in their post about not wanting to tightly couple the Go and Rust since they weren't sure they could accurately represent their structures in both langs.
I don't know anything about this turbo tool, but it seems relatively low-throughput, and they intend to migrate it all to Rust anyway, so it's probably not a big deal.
That said; I'd have done this with binary comms and not JSON, ProtoBuf or not.
For example, it prioritizes backwards and forwards compatibility -- which is not a concern for IPC where you control both ends. So no real optional or required fields. Comparing structures (Go) is awkward and uses reflection. Structs embed a mutex and preserve bytes for unknown fields etc...
MessagePack seems to make a better set of tradeoffs by comparison.
In my case, I was thinking about storage costs for JSON data; but thinking about this use case: Isn’t it true that the CPU would have to spend basically three times as much time making FFI calls? Assuming that practically speaking, field names make up 2/3rds of the bytes in their real-world JSON data.
Though, I know that another team at the same company loved it, but they were all in on java (and groovy and kotlin). As I understand it protobuf meshes with Java's type system much better (and probably Go's too?).
---
I did some benchmarking on various formats/rust libraries when we were deciding on what the protocol should be used for ^ and, IIRC, JSON fared better than you'd have expected. My assumption is that JSON just has had way more eyes/hands on the implementation than anything else. The things that I can remember beating it were (1) bincode, (2) cbor, and (3) protobuf, then JSON was #4.
In retrospect I wish we'd have gone with bincode, but we weren't sure if we were going to need some java code to interact with this service at the time and, at least at the time, bincode was rust-only (and possibly not even stable between compiler releases?). It would have been much faster to develop with and there was a whole bunch of overhead from protobuf that didn't really do anything for us since we were talking over a unix domain socket within a single system.
Whether the codegen/libraries for a particular language provides a more idiomatic binding for these well-known wrappers is up to the implementation - for example, golang libraries have conveniences added for well known libraries: https://pkg.go.dev/google.golang.org/protobuf/types/known. Rust libraries may have the same; I'm not as familiar with the ecosystem there.
JSON is awful in every way.
An idiomatic Go structure does not map well to an idiomatic rust structure, and similarly other languages.
But it's good enough and has very wide and mature support for many languages and a good story for backward compatibility which is important when you have files you want to read 2 years from now or when you have different version of your clients and servers co-existing.
There _could_ be a better interchange format in theory (and many have been proposed), but a combination of maturity and network effect (and the fact that many of the alternatives to protobuf do improve some things but do other things worse than protobuf) make it very hard to chose an alternative.
EDIT: ah, about discriminating zero value from unset value (aka options), they're back in proto3 (experimental) https://github.com/protocolbuffers/protobuf/blob/main/docs/i...
There’s been a lot of discussion of the marshaling and unmarshaling costs of Json (and really any other text format). Should we be looking more strongly at binary formats, and if those have better outcomes than Json?
I’m guessing they didn’t want to introduce more change to the Go code than necessary, but I’m wondering if people have any horror stories to share about any of the binary formats available?