Re:Programmer's critique of missing structure of operating s...
by GoblinSlayer on Wednesday February 19, 2020 @10:08AM
(#22366066)
The disk is a block device, i.e. it reads and writes data in blocks, the system just abstracts that into a stream interface.
Re:Programmer's critique of missing structure of operating s...
by zaarn on Wednesday February 19, 2020 @11:24AM
(#22366953)
Yeah and that is IMO the wrong abstraction. Blocks on disk should be organized into records, where multiple blocks can belong to a single or multiple (if using dedup or Copy-on-Write) records. Implementing a performant stream over that can be done in libc with minimal overhead.
Re:Programmer's critique of missing structure of operating s...
by GoblinSlayer on Thursday February 20, 2020 @03:33AM
(#22373612)
Now you want a different block structure.
Re:Programmer's critique of missing structure of operating s...
by zaarn on Thursday February 20, 2020 @05:25AM
(#22374052)
No? ZFS already organizes blocks on disks this way to some extend and Btrfs uses extends which include multiple blocks together and can be referenced multiple times.
Re:Programmer's critique of missing structure of operating s...
by imtringued on Wednesday February 19, 2020 @11:13AM
(#22366812)
The problem with the TCP vs UDP debate is that there exists a middle ground that satisfies most people's requirements.
TCP offers a single reliable globally ordered stream.
UDP offers unreliable packets in arbitrary order.
What people usually want is to send whole packets(or multiple streams) in arbitrary order with configurable reliability.
TCP and UDP offer neither but UDP has the least restrictions which makes it the only protocol on which you can implement your custom protocol.
Re:Programmer's critique of missing structure of operating s...
by bvrmn on Wednesday February 19, 2020 @10:38AM
(#22366396)
> Implementing streaming files over record-based files is easier
What a performance-wise nightmare. Goodbye page cache and other zero copy optimizations.
Re:Programmer's critique of missing structure of operating s...
by zaarn on Wednesday February 19, 2020 @11:22AM
(#22366936)
Not really, the OS still has to abstract it onto the harddisk, this should be trivial to achieve without impacting page cache and zero copy optimizations if the on-disk metadata is seperated sufficiently. COW Filesystems already do it to a lesser degree and they're certainly fine.
Re:Programmer's critique of missing structure of operating s...
by bvrmn on Wednesday February 19, 2020 @11:41AM
(#22367167)
What to do with clients who want a stream of records? Separated meta + payload allows fast byte streams. Combined meta + payload allows fast record streams. Structured implementation can't provide performance for both cases. But byte-oriented can. And we see this approach in modern and fast OSes.
Re:Programmer's critique of missing structure of operating s...
by zaarn on Wednesday February 19, 2020 @11:43AM
(#22367197)
Allow clients to choose which they receive?
Re:Programmer's critique of missing structure of operating s...
by bvrmn on Wednesday February 19, 2020 @11:56AM
(#22367351)
May be I was unclear. If we introduce record abstraction for file ops then OS low-level implementation have to decide which way to store structured data. meta + payload or meta separated from payload. In latter case there is no way to provide fast record streams. Former case does not allow fast byte streams. Client decision to receive bytes or records doesn't magically zero copy reorder stored data.
Re:Programmer's critique of missing structure of operating s...
by zaarn on Wednesday February 19, 2020 @11:59AM
(#22367398)
Why would the later cause record streams to be slow? Snapshots in COW FS' aren't that terribly slow either, even if there is quite a few of them, which basically boils down to the same structure, really.
Lots of people implement half of TCP over UDP because they want packet based transmission but also care about losing packets.
Implementing packets on top of a stream is not as trivial as implementing a stream over packets.
Implementing streaming files over record-based files is easier, you just iterate over each record and append to the stream. Implementing records in stream-based files means you have to define a framing format, internal format, bookkeeping, etc.
Record-based files would be the proper lego building blocks.