News for hackers. Stuff that matters.
Re:How Turborepo is porting from Go to Rust by bluejekyll on Friday July 21, 2023 @11:24AM (#36814897)
> By serializing to JSON, a format with robust support in both Go and Rust, we were able to minimize our FFI surface area, and avoid a whole class of cross-platform, cross-language bugs.

There’s been a lot of discussion of the marshaling and unmarshaling costs of Json (and really any other text format). Should we be looking more strongly at binary formats, and if those have better outcomes than Json?

I’m guessing they didn’t want to introduce more change to the Go code than necessary, but I’m wondering if people have any horror stories to share about any of the binary formats available?

Re:How Turborepo is porting from Go to Rust by wrs on Friday July 21, 2023 @11:32AM (#36814987)
It depends on what you’re marshaling, and how much. Anyway, in my experience, the biggest practical disadvantage of JSON is not performance, it’s that people tend not to put a schema validation/migration layer on top of it, which leads over time to chaos. Many of the popular binary formats have schema validation built in.
Re:How Turborepo is porting from Go to Rust by throwawaymaths on Friday July 21, 2023 @11:44AM (#36815164)
Totally insane that people don't put validation since jsonschema is quite nice
Re:How Turborepo is porting from Go to Rust by packetlost on Friday July 21, 2023 @11:54AM (#36815336)
Text formats have innate issues when it comes to deserialization: you don't know the size of each field of your data until you scan the entire field. This makes deserializers generally more prone to DOS attacks. Many binary formats must necessarily store the size of fields as metadata, which means you might not need to deserialize the whole message to interpret parts of it. Further, JSON doesn't support streaming natively. These are mostly unique use-cases with marginal improvements, but it can matter.
Re:How Turborepo is porting from Go to Rust by importantbrian on Friday July 21, 2023 @11:36AM (#36815035)
Isn't this the point of things like protobuf and thrift? It seems like everyone beyond a certain scale sends data around in some binary format rather than JSON.
Re:How Turborepo is porting from Go to Rust by bluejekyll on Friday July 21, 2023 @11:45AM (#36815194)
Yes, I’m wondering if people have bad experiences with CBOR, thrift, flatbuffers, protobuf, messagepack, etc. There are a huge number. I think Protobuf might be the best choice across all languages at this point, maybe?

But that’s my question, if you choose a binary format, what are the issues to look out for?

Example, I work with Java, Rust, Go, and Python at work. I have specifically had issues with Avro (not my choice) which is really well supported in Java, but not much else.

Re:How Turborepo is porting from Go to Rust by f_devd on Friday July 21, 2023 @11:54AM (#36815338)
I dislike protobuf for the reason that the generated code is often quite bad (in terms of the interface in the language you are using) and in external projects it's often a manual step (sometimes with additional headache of protoc2 vs protoc3) to just build the project. In terms of message specification it's quite good though.
Re:How Turborepo is porting from Go to Rust by packetlost on Friday July 21, 2023 @11:59AM (#36815409)
This is the reason we opted for building our own code gen system instead of Thrift or Protobuf: they both generate really ugly code in the languages we wanted to use
Re:How Turborepo is porting from Go to Rust by mdtusz on Friday July 21, 2023 @12:19PM (#36815695)
Still early days, but we've been using CBOR instead of JSON lately at work for interfaces that have "settled" and it's been great. Means that you can shake out the early integration issues using human readable JSON, then just switch the ser/de once it's all playing nice.

Binary data support is pretty nice too for avoiding multipart request bodies.

Re:How Turborepo is porting from Go to Rust by importantbrian on Friday July 21, 2023 @02:03PM (#36817220)
Ah, I see what you were saying now. Yeah, I don't have direct experience with those binary formats so I can't say. I'm not an Avro fan either but there are other binary formats like parquet that are easier to work with and at least in my corner of the tech world seems to have become the de facto standard.
Re:How Turborepo is porting from Go to Rust by alias_neo on Friday July 21, 2023 @11:51AM (#36815283)
I would have suggested ProtoBuf too.

That said, there was a comment in their post about not wanting to tightly couple the Go and Rust since they weren't sure they could accurately represent their structures in both langs.

I don't know anything about this turbo tool, but it seems relatively low-throughput, and they intend to migrate it all to Rust anyway, so it's probably not a big deal.

That said; I'd have done this with binary comms and not JSON, ProtoBuf or not.

Re:How Turborepo is porting from Go to Rust by iamcalledrob on Friday July 21, 2023 @12:21PM (#36815737)
I'm using protobuf for exactly this use-case, but I'd say that it's clearly designed as a network format first, and makes trade-offs as such.

For example, it prioritizes backwards and forwards compatibility -- which is not a concern for IPC where you control both ends. So no real optional or required fields. Comparing structures (Go) is awkward and uses reflection. Structs embed a mutex and preserve bytes for unknown fields etc...

MessagePack seems to make a better set of tradeoffs by comparison.

Re:How Turborepo is porting from Go to Rust by jamil7 on Friday July 21, 2023 @11:49AM (#36815256)
They don't specify what types of issues they had with FFI which makes me somewhat dubious of the outcome here. There are tools to help paper over the rough edges of calling code through FFI, at least with Rust. Maybe the reasoning is more to do with the Go side.
Re:How Turborepo is porting from Go to Rust by bluejekyll on Friday July 21, 2023 @05:07PM (#36819764)
From my experience, the issue is generally related to passing complex data structures across FFI boundaries. For this, adopting a serialization format is nicer than say deconstructing each object into proper parameters, which really end up being very similar code to that which is generated with the serialization libraries.
Re:How Turborepo is porting from Go to Rust by adameasterling on Friday July 21, 2023 @11:59AM (#36815404)
A few months ago, I did some analysis on JSON data in our platform, and discovered that more than two-thirds of bytes were field names. Two thirds! That’s a lot of potentially unnecessary extra bytes.

In my case, I was thinking about storage costs for JSON data; but thinking about this use case: Isn’t it true that the CPU would have to spend basically three times as much time making FFI calls? Assuming that practically speaking, field names make up 2/3rds of the bytes in their real-world JSON data.

Re:How Turborepo is porting from Go to Rust by evntdrvn on Friday July 21, 2023 @12:29PM (#36815848)
JSON compresses very well, fortunately
Re:How Turborepo is porting from Go to Rust by azdle on Friday July 21, 2023 @12:00PM (#36815425)
Protobuf is incredibly annoying to use with rust (my usecase was specifically rust to-rust) because of how limited protobuf's type system is. For instance there is no way to represent an `Option` type because protobuf has decided that if something is no present, then it is really some default value instead. The only workaround I found was to have both `foo` and `is_fo_set` fields, which required having duplicate copies of every datatype one for (de)serialization and one for actually using in my codebase, manually implementing changes between them. Don't get me wrong, there's nothing show-stopping, but it was definitely a death by a thousand cuts situation for me.

Though, I know that another team at the same company loved it, but they were all in on java (and groovy and kotlin). As I understand it protobuf meshes with Java's type system much better (and probably Go's too?).

---

I did some benchmarking on various formats/rust libraries when we were deciding on what the protocol should be used for ^ and, IIRC, JSON fared better than you'd have expected. My assumption is that JSON just has had way more eyes/hands on the implementation than anything else. The things that I can remember beating it were (1) bincode, (2) cbor, and (3) protobuf, then JSON was #4.

In retrospect I wish we'd have gone with bincode, but we weren't sure if we were going to need some java code to interact with this service at the time and, at least at the time, bincode was rust-only (and possibly not even stable between compiler releases?). It would have been much faster to develop with and there was a whole bunch of overhead from protobuf that didn't really do anything for us since we were talking over a unix domain socket within a single system.

Re:How Turborepo is porting from Go to Rust by bminor13 on Friday July 21, 2023 @12:26PM (#36815809)
proto3 messsage fields allow for detecting set vs. not set; other field types (repeated, map, int32, string, bool, enum, etc.) have this "default value if not set" issue. The canonical way of handling this is to use wrapper messages (because one can detect if the wrapper is not set), and there are "well-known" canned messages/protos one can import and use without writing their own: https://protobuf.dev/reference/protobuf/google.protobuf/

Whether the codegen/libraries for a particular language provides a more idiomatic binding for these well-known wrappers is up to the implementation - for example, golang libraries have conveniences added for well known libraries: https://pkg.go.dev/google.golang.org/protobuf/types/known. Rust libraries may have the same; I'm not as familiar with the ecosystem there.

Re:How Turborepo is porting from Go to Rust by square_usual on Friday July 21, 2023 @12:25PM (#36815790)
Have you tried FlatBuffers? And if you did, did it perform worse than JSON? The idea of deserialize-on-read is tempting to me, but I'd like to see real-world cases of it working out.
Re:How Turborepo is porting from Go to Rust by ddorian43 on Friday July 21, 2023 @04:21PM (#36819149)
FlatBuffers should be faster than protobuff.
Re:How Turborepo is porting from Go to Rust by rvcdbn on Friday July 21, 2023 @01:17PM (#36816593)
recent versions of proto3 have added back the “optional” keyword that can be used on any field. see: https://github.com/protocolbuffers/protobuf/blob/main/docs/f...
Re:How Turborepo is porting from Go to Rust by foobiekr on Friday July 21, 2023 @01:39PM (#36816888)
On optional.. this was a regression in proto that is somewhat helped by https://github.com/protocolbuffers/protobuf/blob/main/docs/f... ; I have no idea whether protobuf for rust has started taking advantage of this.

JSON is awful in every way.

Re:How Turborepo is porting from Go to Rust by athrowaway3z on Friday July 21, 2023 @02:41PM (#36817746)
iirc https://github.com/tokio-rs/prost has the proper proto3 'optional' -> Option<T> support
Re:How Turborepo is porting from Go to Rust by ithkuil on Friday July 21, 2023 @06:01PM (#36820503)
Yes it is unsurprising that an IDL designed to be used as an interchange format between different languages (and other requirements such as backward compatibility etc) doesn't perfectly match your specific language type system.

An idiomatic Go structure does not map well to an idiomatic rust structure, and similarly other languages.

Re:How Turborepo is porting from Go to Rust by rattray on Friday July 21, 2023 @08:03PM (#36821709)
To be fair, protobuf is a pretty poor/annoying match to many languages that aren't Go or Java, including TS. Optional types, for example, are very much not unique to Rust.
Re:How Turborepo is porting from Go to Rust by ithkuil on Friday July 21, 2023 @08:17PM (#36821821)
Don't get me wrong, it is a PITA.

But it's good enough and has very wide and mature support for many languages and a good story for backward compatibility which is important when you have files you want to read 2 years from now or when you have different version of your clients and servers co-existing.

There _could_ be a better interchange format in theory (and many have been proposed), but a combination of maturity and network effect (and the fact that many of the alternatives to protobuf do improve some things but do other things worse than protobuf) make it very hard to chose an alternative.

EDIT: ah, about discriminating zero value from unset value (aka options), they're back in proto3 (experimental) https://github.com/protocolbuffers/protobuf/blob/main/docs/i...