News for hackers. Stuff that matters.
Re:Programmer's critique of missing structure of operating s... by tyingq on Wednesday February 19, 2020 @08:21AM (#22365169)
"The database for a filesystem is an classic. WinFS, the filesystem that should have been a key feature of Longhorn/Windows Vista is based on a relational database."

I always wondered why Unix never added record-based files, in addition to stream-based files...like mainframes have. That would have simplified many things.

Re:Programmer's critique of missing structure of operating s... by bregma on Wednesday February 19, 2020 @08:57AM (#22365425)
> I always wondered why Unix never added record-based files, in addition to stream-based files...like mainframes have. That would have simplified many things.

You can implement record-based files (RAX, ISAM, etc) on top of stream-based files. The Unix philosophy was like Lego: small parts that do simple things that can be put together in combinations to build greater things. If you can build it from the basic blocks, it's not a basic block.

There are plenty of more advanced storage systems available for Unix. You don't need the OS vendor to supply The One.

Re:Programmer's critique of missing structure of operating s... by imtringued on Wednesday February 19, 2020 @11:05AM (#22366716)
>If you can build it from the basic blocks, it's not a basic block.

The idea behind the author's post is that a directed acyclic graph is a basic block that is just one layer of abstraction above raw bytes. JSON, YAML, TOML, INI, .properties, environment variables are all just different ways to represent a graph and if it were possible to store the graph directly in the filesystem then one would not have to manually [0] serialize the graph to bytes and deserialize it back later.

[0] graph -> byte conversion is the job of the graph filesystem

Re:Programmer's critique of missing structure of operating s... by Bystroushaak on Wednesday February 19, 2020 @01:15PM (#22368292)
Thank you for pointing that out, thats exactly one of my points.
Re:Programmer's critique of missing structure of operating s... by zaarn on Wednesday February 19, 2020 @09:17AM (#22365568)
The problem is triviality to implement, a similar problem occurs with TCP and UDP.

Lots of people implement half of TCP over UDP because they want packet based transmission but also care about losing packets.

Implementing packets on top of a stream is not as trivial as implementing a stream over packets.

Implementing streaming files over record-based files is easier, you just iterate over each record and append to the stream. Implementing records in stream-based files means you have to define a framing format, internal format, bookkeeping, etc.

Record-based files would be the proper lego building blocks.

Re:Programmer's critique of missing structure of operating s... by GoblinSlayer on Wednesday February 19, 2020 @10:08AM (#22366066)
The disk is a block device, i.e. it reads and writes data in blocks, the system just abstracts that into a stream interface.
Re:Programmer's critique of missing structure of operating s... by zaarn on Wednesday February 19, 2020 @11:24AM (#22366953)
Yeah and that is IMO the wrong abstraction. Blocks on disk should be organized into records, where multiple blocks can belong to a single or multiple (if using dedup or Copy-on-Write) records. Implementing a performant stream over that can be done in libc with minimal overhead.
Re:Programmer's critique of missing structure of operating s... by GoblinSlayer on Thursday February 20, 2020 @03:33AM (#22373612)
Now you want a different block structure.
Re:Programmer's critique of missing structure of operating s... by zaarn on Thursday February 20, 2020 @05:25AM (#22374052)
No? ZFS already organizes blocks on disks this way to some extend and Btrfs uses extends which include multiple blocks together and can be referenced multiple times.
Re:Programmer's critique of missing structure of operating s... by imtringued on Wednesday February 19, 2020 @11:13AM (#22366812)
The problem with the TCP vs UDP debate is that there exists a middle ground that satisfies most people's requirements.

TCP offers a single reliable globally ordered stream.

UDP offers unreliable packets in arbitrary order.

What people usually want is to send whole packets(or multiple streams) in arbitrary order with configurable reliability.

TCP and UDP offer neither but UDP has the least restrictions which makes it the only protocol on which you can implement your custom protocol.

Re:Programmer's critique of missing structure of operating s... by bvrmn on Wednesday February 19, 2020 @10:38AM (#22366396)
> Implementing streaming files over record-based files is easier

What a performance-wise nightmare. Goodbye page cache and other zero copy optimizations.

Re:Programmer's critique of missing structure of operating s... by zaarn on Wednesday February 19, 2020 @11:22AM (#22366936)
Not really, the OS still has to abstract it onto the harddisk, this should be trivial to achieve without impacting page cache and zero copy optimizations if the on-disk metadata is seperated sufficiently. COW Filesystems already do it to a lesser degree and they're certainly fine.
Re:Programmer's critique of missing structure of operating s... by bvrmn on Wednesday February 19, 2020 @11:41AM (#22367167)
What to do with clients who want a stream of records? Separated meta + payload allows fast byte streams. Combined meta + payload allows fast record streams. Structured implementation can't provide performance for both cases. But byte-oriented can. And we see this approach in modern and fast OSes.
Re:Programmer's critique of missing structure of operating s... by zaarn on Wednesday February 19, 2020 @11:43AM (#22367197)
Allow clients to choose which they receive?
Re:Programmer's critique of missing structure of operating s... by bvrmn on Wednesday February 19, 2020 @11:56AM (#22367351)
May be I was unclear. If we introduce record abstraction for file ops then OS low-level implementation have to decide which way to store structured data. meta + payload or meta separated from payload. In latter case there is no way to provide fast record streams. Former case does not allow fast byte streams. Client decision to receive bytes or records doesn't magically zero copy reorder stored data.
Re:Programmer's critique of missing structure of operating s... by zaarn on Wednesday February 19, 2020 @11:59AM (#22367398)
Why would the later cause record streams to be slow? Snapshots in COW FS' aren't that terribly slow either, even if there is quite a few of them, which basically boils down to the same structure, really.
Re:Programmer's critique of missing structure of operating s... by tyingq on Wednesday February 19, 2020 @10:07AM (#22366061)
"You can implement record-based files (RAX, ISAM, etc) on top of stream-based files"

Sure. And we have, many times over. A base implementation that supported fixed and variable records, shared process mmap() access, etc...would have value. And perhaps the kernel being aware of it would have some benefit.

Re:Programmer's critique of missing structure of operating s... by pdimitar on Saturday February 22, 2020 @08:40AM (#22390927)
> The Unix philosophy was like Lego: small parts that do simple things that can be put together in combinations to build greater things. If you can build it from the basic blocks, it's not a basic block.

And the normal programmer's logic is: "if it's not already integrated in the OS then I am not going to use it". So stream files it is, forever.

I think history has proven this. People are very risk-averse even if the risk is quite minor (example: using sqlite3 instead of the file system).