Serde generates heaps and heaps of generic code. This all gets optimized away to be very efficient, but only once it reaches LLVM.
Ever tried working on a crate with hundreds or thousands of de/serializable types? Compile times shoot through the roof really quickly, and serde is often the culprit.
The maintainer of serde also created `miniserde` [1] to tackle this problem, which uses dynamic dispatch instead of generics and can have 4x compile time improvements.
Due to Rusts lack of orphan instances you really depend on a pervasive standard for serialization, though, so the ecosystem is pretty locked in to serde.
For example you can pre-instatiate the most common type parameters and just stuck those instantiations into a binary library.
Also if you have a nice C++ compiler environment like Visual C++, you can even do edit-and-continue for many kinds of changes, and VS 2022 will double down on that capability.
Whatever the solution Rust comes up with I think will look at the practical component so as to actually see adoption. It seems like the Rust team is very good at studying the mistakes other languages make in their approaches to a problem and finding the right fit within the language/targeted domain.
With regards to your specific experience, I don't know that I'd say that Clearcase, HP-UX or aCC are main-stream professional C++ development environments. I think for that you're really looking at clang, gcc, & msvc with icc being a distant fourth that has itself moved to clang recently. The primary target OS would either be server Linux, Android, macOS, iOS with maybe FreeBSD picking up the slack. HP-UX has a niche but it's a small niche comparatively in terms of $ spent on developers in that ecosystem. In terms of source control, everyone is mostly on git or mercurial. Anyone not just refuses to get with the times because they have bought into their existing tool expertise but don't know how to adapt.
The compile time costs for the error handling crates is due to increase in build graph size, not due to the cost of running the error handling proc macros or the cost of compiling the code generated from them. For anything but a small project, that increase in graph size is probably not a big deal (and likely the project is already using much of the same deps). This cost does not increase as a project gets larger.
serde also probably has the build graph size problem, but the bigger problem is the cost generating and compiling the generated code. This cost tends to increase as a project gets larger.
Compilation speed isn't just one number. While 5min isn't bad at all for compiling a whole project from scratch including dependencies, 5min is a really long time for incremental builds.
I'm not sure why. The fact that Rust makes dyn traits significantly more verbose doesn't help, of course, but that can't be the only reason.
The main problem is that trait objects are very limited.
There is no downcasting or upcasting (eg C++ dynamic_cast) and trait objects are limited to a single trait. You can't have `Box<dyn A + B>`.
That leads to lots of headaches in practice.
I hear downcasting might be on the horizon, so that's something to look forward to.
Is there any actual reason something like serde couldn't exist in other languages? None that I know of, but it doesn't, at least not when you consider just how pervasive serde is in the ecosystem (almost every library that stores things you want to serialize will have it). I really hope every language can standardize on something similar; it just makes development so much nicer and faster when you don't have to worry about this stuff.
This has not been remotely my experience. Deserializing has a lot of implications for ownership and you get into complicated deserializer trait specifications pretty quickly.
In particular, there doesn’t seem to be a way to deserialize a type that has a reference because you don’t seem to be able to tell serde who should own that memory. Maybe I’m wrong, but I’ve walked through this multiple times with experienced Rust users and no one could get it working.
Serde is impressive, I’m sure, but far from “don’t even have to think about it”.
We have hit one serde footgun, which is that skip-serializing-if corrupts your data with formats like bincode :-(. https://github.com/serde-rs/serde/issues/1732 really wish that could be fixed.
C# standard library has DataContractSerializer for decades. That thing reads/write text XML by default, but with minor tweaks can do binary XML or JSON.
I like their binary XML format the most. Very fast because doesn’t waste time printing or parsing numbers or Base64 bytes. Also, pre-shared XML dictionary, and session-accumulating dynamic dictionary, makes the serialized representation very compact, sometimes an order of magnitude better than text/xml.
That said, it's incredibly refreshing to be able to focus on these semantic and migration concerns without having to worry about how to deal with parsing them later. serde doesn't solve enforcing backwards compatibility for you, but it sure makes doing the right thing a lot easier. You never have to worry about some code accidentally forgetting to check the version field, people carelessly mixing and matching messages with different versions, or fields that "should" never be set but are there for backwards compatibility. That's worth a lot to me!
public record Point(int x, int y) {}
// [...]
var objectMapper = new ObjectMapper();
var jsonString = objectMapper.writeValueAsString(new Point(2,1));
Manifold [1] an example framework that relies heavily on that, achieving similar things like Serde.
Java developers don't like to use these things too much though, unlike Rust developers who love their `#derive`, so you're mostly right that reflection-based serialization is still more common, but that's by choice.
Dependency Injection is in a similar situation: frameworks like Micronaut [2] can do it without reflection, and things like Google Dagger [3] have existed for several years that do the same thing... but still, most Spring Boot-based projects I know of use reflection-based DI as well... it's fast and the security issues have been largely mitigated nowadays.
[2] https://docs.micronaut.io/latest/guide/
[3] https://rskupnik.github.io/dependency-injection-in-pet-proje...
That's still (at least naively) using reflection, but it's a lot safer.
In practice, using reflection for every object being deserialised is far too slow and Jackson at least generates code at runtime rather than repeatedly reflecting. That's magic, but it's nice magic because it's not breaking the language: one can see that it's possible to implement it in pure Java.
Reflection is indeed used there, but not to serialize. It's only used once per type, to generate code.
C# has multiple ways to generate code in runtime. One is Reflection.Emit, allows to manually generate bytecode instructions + metadata. For instance, one can build new types in runtime. A typical pattern is implementing manually-written abstract class or interface with generated types: this way the manually-written code can call into runtime generated one.
Another method is System.Linq.Expressions. This one allows to generate code (no new types though, just functions) from expression trees, and provides API to build and transform these expression trees.
Regardless on the method, the generated code is no different from manually written code. JIT compiler can even inline things across, when generated code calls manually written one, or vice versa.
There's a weird (not in a bad way) elegance here that reminds me of lisp.
Yes and no.
No because when you try to do unsupported things like calling a method on an object which doesn’t support one, you gonna get an appropriate runtime exception.
Yes because if you fail lower-level things like local parameter allocation, you gonna get an appropriate runtime exception but that one is (1) too late, I’d prefer such things to be detected when you emit the code, not when trying to use the generated code (2) Lacks the context.
Overall, when I can I’m using that higher-level System.Linq.Expressions for runtime codegen. Things are much nicer at that level. I only using the low-level thing when I need to emit new types, like there: https://github.com/Const-me/ComLightInterop/blob/master/ComL...
So at compile time, just like serde works, you create your type safe serializer.
A generic JSON decoder would give you a dynamic structure that can contain anything, and then you'd have to pick it apart.
Serde just doesn't use reflection to do it.
Performance of a lot of the reflection-based parsing APIs also leaves something to be desired, which means that projects with more stringent performance requirements often have to resort to stuff like Avro. This still happens with serde, but much more rarely--besides having more compile-time information at its disposal and needing to allocate less, it also provides relatively straightforward hooks to achieve things like zero-copy deserialization for strings where the whole buffer is available at once.
How does Serde work with optional extra fields? I know you can use Option<>, but that implies you know they exist - what happens if fields get added in the future silently? I guess the schema changes then, which isn't good, but that might happen?
So you can define a struct with one status field and the right annotations and you will get exactly what you describe without having to write the code and it will still be almost as fast as doing that parsing yourself.
This also informs your second question. If new optional fields in json are added, unless you tell serde to complain about them, it won't, and you can add the new Option at your leisure.
#[derive(Deserialize)]
struct Response { status: u64 }You can define a struct with just the status field.
> How does Serde work with optional extra fields? I know you can use Option<>, but that implies you know they exist - what happens if fields get added in the future silently? I guess the schema changes then, which isn't good, but that might happen?
Unknown fields are skipped by default.
For JS and Python, JSON.stringify/parse and json.loads/dumps are a step up, but you still end up with an untyped mess with no schema validation, which makes them only halfway solutions to me. I'm a static typing guy at heart, sue me.
Let's take JSON or YAML for example:
Rust's closest siblings C and C++ are typed, and you get a pretty much untyped messes when working with serialisation or deserialisation. You have to manually inspect every node or write manual ”NodeType" to struct conversation. They allow unwrapping to primitives at best (eg via template specialisation).
In Haskell it's a bit more awkward, but you can kinda pull something similar off in terms of ergonomics. I've not worked with JSON in Haskell that much that I need to look for it, but I don't remember there being something as ergonomic as serde.
Typescript doesn't support this type of validation either out of the box (there might be tools that add validation).
In Elixir there's the Poison library, or it could be done with Kernel.struct/2 (not 100% sure this will work though).
Of the "mainstream" languages Go is pretty much the only one that has as good ergonomics as Rust in that it supports it pretty much natively (via struct tags)
---
This list excludes codegen tools which can generate (de)serialisers from an external schema (a la capnp, grpc, jsonschema, etc).
I think Haskell's most popular JSON library aeson is basically the same thing and probably a source of inspiration for serde.
data MyCustomType = ... deriving (Generic, ToJSON, FromJSON)
-- I prefer snake case instead
data MyCustomType = ...
deriving Generic
deriving (FromJSON, ToJSON)
via CustomJSON '[FieldLabelModifier CamelToSnake]Swift does this too, using the Codable Protocol. The compiler generates the protocol conformance code for you, if you say a type is `Codable`, and the properties the type contains are all Codable types (Int, Float, String, URL, etc. conform to it), you don't have to do anything.
https://developer.apple.com/documentation/foundation/archive...
(1): which is a shame, I remember when it came out that it seemed like a nice language
var account = JsonConvert.DeserializeObject<Account>(string/stream)
That's it. No inspection or looping through trees. You can often get it to create objects via constructor for validation.
It's been like this for 15 years...
I've worked with the Microsoft tech stack very very little, so I've never bothered to learn C#.
I've completely forgot about Java; I haven't used it in ages. Now that you mentioned Java, I realised Kotlin has it too.
I've done some benchmarking on JSON -> in-memory borrow structs and it was pretty darn impressive while being fairly ergonomic to use compared to a streaming parser.
I've dealt with a number of C++ based serialization libraries, and they always have serious downsides to the point I often end up hand serializing structures with one-off half-baked formats just like you, which is error prone and laborious.
It is more convenient to use serde to serialize/deserialize to some standard format like JSON or YAML than it is to write your own half-baked format and serialization/deserialization code --- even for the simplest tasks --- so you just stop doing the latter. This is a fundamental shift, and very good for your software.
"Convenience" here actually covers a lot of ground. The APIs are convenient. Boilerplate code is minimal (#[derive(Serialize)]). serde's attributes give you lots of control over the serialization/deserialization, while still being convenient to use. cargo makes importing serde into your project effortless. All of these are necessary to achieve that shift away from custom formats.
Before Rust I used to write a lot of one-off half-baked formats. Pernosco uses Rust and serde and it uses no half-baked formats. Everything's JSON, or YAML if it needs to be more human-editable, or bincode if it needs to be fast and compact and not human-readable or extensible.