News for hackers. Stuff that matters.
Re:Ron: Rusty Object Notation by amelius on Sunday April 23, 2023 @06:01AM (#35674878)
Why can't we just save Rust data-structures in a memory-mapped file? That would make saving and loading much more efficient, and it would also make programming much more convenient.
Re:Ron: Rusty Object Notation by tialaramex on Sunday April 23, 2023 @06:20AM (#35674971)
You can do this, but only if you know exactly what that state means. In practice this is almost certainly too fragile to be useful. If you change anything you probably invalidate your understanding of this data and cause chaos.

Rust's in memory data structures don't have any meaning on their own, they only have meaning in the context of the running program.

This is, in fact, the whole point of the New Type Idiom. If I have two types Minutes(u32) and Years(u32) those are represented in memory the same way, an unsigned 32-bit integer, just four bytes. But because Rust is statically Type Checked, everywhere the programs knows which is which, you can't assign a Minutes to a Years, or a Years to a Minutes, for the same reason you can't assign "Hacker News" to an f64.

But this means a 32-bit value in your hypothetical file could be either a Minutes or a Years, and a program loading that file needs to know a priori which it is. Once you start coming up with rules about how you'd do that, you're inventing a serialisation format, and you're exactly back to the problem where your original question arose.

Re:Ron: Rusty Object Notation by capableweb on Sunday April 23, 2023 @06:46AM (#35675108)
> You can do this, but only if you know exactly what that state means. In practice this is almost certainly too fragile to be useful. If you change anything you probably invalidate your understanding of this data and cause chaos.

I've used the crate `savefile` for this in the past (https://docs.rs/savefile/latest/savefile/), which also supports migrating data between versions. So if you have a bunch of on-disk data of a struct and you add a new property, you can handle migrating the data on the fly to the new struct structure.

I'm also using `savefile` for saving HN items (comments and stories) to disk in my own HN application: https://codeberg.org/ditzes/ditzes/src/branch/master/src-tau... and it's very quick to load stuff from disk, haven't hit any bottlenecks really

Example: https://ditzes.com/item/35640002 (SpaceX Starship rocket explodes minutes after launch from Texas) has 1223 comments which are individually loaded from disk, takes around 100ms to load from disk, which considering how inefficient and unoptimized the code is, I'm pretty happy with.

Re:Ron: Rusty Object Notation by palunon on Sunday April 23, 2023 @07:28AM (#35675311)
> Once you start coming up with rules about how you'd do that, you're inventing a serialisation format, and you're exactly back to the problem where your original question arose.

Which is exactly what savefile does.

Re:Ron: Rusty Object Notation by amelius on Sunday April 23, 2023 @09:02AM (#35675884)
Sorry, but I think you are turning small hurdles into roadblocks in your mind.

It is certainly possible to store data structures directly inside files (I think MS Word did this to save documents), but indeed you have to think carefully about type safety but also data alignment, offsetting and such, and perhaps cryptographic signatures to make it secure when needed, which is what library/language designers need to think about to make it easier for programmers to use this stuff. Sure, it won't be adequate in all situations, but that's not what I'm asking for.

Re:Ron: Rusty Object Notation by habitue on Sunday April 23, 2023 @06:04AM (#35674895)
I mean, you can, if there are no pointers. But then you still need to verify the layout when you reload it etc.

Once you solve the various edge cases, this is basically what flatbuffers and capnproto are

Re:Ron: Rusty Object Notation by j-pb on Sunday April 23, 2023 @06:05AM (#35674901)
The information what datastructures are in memory is based on compile time static analysis data, you wouldn't know what you're loading without some additional data annotations. It would also require unreasonably fixing memory layout implementation details between versions.
Re:Ron: Rusty Object Notation by amelius on Sunday April 23, 2023 @10:01AM (#35676303)
But these are things Rust can solve for the programmer! It can add annotations to the data and even provide versioning.
Re:Ron: Rusty Object Notation by 0dayz on Sunday April 23, 2023 @06:04AM (#35674892)
I would encourage you to try and make that idea happen.
Re:Ron: Rusty Object Notation by Findecanor on Sunday April 23, 2023 @08:22AM (#35675647)
Besides the issues that others have mentioned (typing, storing nested data-structures, pointers), you would add a requirement on the backing OS and file system to enforce Mandatory File Locking.

While Windows enforces it, most Unix systems don't support it at all. Back when Linux had it (removed in 2021), it had to be enabled when a volume was mounted, and that was not default nor was it supported for all file system types.

Without it, even if you mmap() the file with the MAP_PRIVATE flag, portions of the file could be overwritten by other programs before you read them.

Re:Ron: Rusty Object Notation by mort96 on Sunday April 23, 2023 @09:34AM (#35676103)
People like to send data around between different computers. Different computers have different endianness and potentially different padding requirements and differently sized 'usize's.

And there are the challenges with pointers and data structures others have mentioned. And supporting old files in the event of a Rust ABI change becomes incredibly hard.

Re:Ron: Rusty Object Notation by kortex on Sunday April 23, 2023 @10:31AM (#35676545)
We (the institution of programmers broadly speaking) tried that, and it was a hot mess. It's fine for scratch for a single machine, running a single version of the software. As soon as you involve multiple computers talking or multiple software versions over time, it quickly causes tons of issues.

There's a reason most serialization formats are either self-describing, have external schemas (json), or offload state to a specific state store engine.

Re:Ron: Rusty Object Notation by dmytrish on Sunday April 23, 2023 @01:05PM (#35678090)
It's a shame you're being downvoted, it's a perfectly valid question.

It is possible to do explicitly with `#[repr(C)]` in Rust. This is still fragile:

- `usize` and `isize` are not guaranteed to be portable between machines.

- pointers (including vtables in trait objects) are not portable between processes

- structs and enums may change in source code

- file descriptors, mutexes/futexes and any kind of external resources in general are not portable between processes. Thread-local variables are not even portable between threads.

In short: memory layout in general depends on a large amount of ephemeral compiler metadata. It is possible to constrain memory layout to something portable (protobufs, flatbuffers, capnproto, thrift, etc), but this seems more like a serialization format, not an efficient memory layout for a compiled language.

I'd say this is a good thing to avoid ossification of memory layouts: if you need to unmarshal something, it is better to mark this explicitly. Although the C ABI can be convenient, it constrains optimizations that compilers are allowed to do.