News for hackers. Stuff that matters.
Re:Programmer's critique of missing structure of operating s... by GuB-42 on Wednesday February 19, 2020 @06:17AM (#22364610)
This is mostly a critique of UNIX, but several of these concepts are already implemented in other OSes, either in production or experimentally.

The database for a filesystem is an classic. WinFS, the filesystem that should have been a key feature of Longhorn/Windows Vista is based on a relational database.

The "death of text configuration files" is the idea behind the Windows registry.

Powershell (Windows, again) is based on structured data rather then text streams.

For the "programs as a collection of addressable code blocks", when we think about it, we are almost there. An ELF executable for instance is not just a blob that is loaded in memory. It is a collection of blocks with instructions on how to load them, and it usually involves addressing other blocks in other ELF files (i.e. dynamic libraries), with names as keys. We could imagine splitting the executable apart to be saved in a database-like filesystem, that would work, but if wouldn't change the fundamentals.

The problem I have with structure is that it implies a schema. And without that schema there is nothing you can do. And of course, because we all have different needs, there are going to be a lot of schemas. So now you turn one problem into two problems: manage the schemas and manage the data. With a UNIX-style system, even if you need some kind of structure to actually process the data, the system is designed in such a way that for common operations (ex: copy), you don't need an application-specific schema.

Re:Programmer's critique of missing structure of operating s... by anonsivalley652 on Wednesday February 19, 2020 @06:36AM (#22364700)
Yes, text configuration files are dumb because they require N parser/script editors times M configurable programs. Furthermore, what people really want is universal progmatic and CLI access to configuration.

Microsoft Microsoft'ed configuration management with the way they implemented the registry. The Apple PList way sort-of goes there but doesn't quite master it.

A better way would've been is configuration via code and command-line that's easy to interface for all purposes. It's important to:

- be able to transactionally backup and restore all settings

- have multiple instances of the same program with different settings

- wipe out settings to default

- enumerate all settings within a program or within the whole system

- subscribe and be notified of setting changes

- ACLs or privileges to separate users and processes from each other (similar to the Windows Registry)

- audit and undo setting changes

- hide secrets that are provided from dynamic providers

- allow dynamic providers for values of settings

Re:Programmer's critique of missing structure of operating s... by packet_nerd on Wednesday February 19, 2020 @09:48AM (#22365859)
Switch and router OSes have a unique CLI configuration system that is both simple and quite powerful. Compared to Linux I especially like the ability to enumerate the entire configuration with one command ("show running-config" in Cisco's IOS, for example).

I would love to be able to see and manage a Linux system's configuration in an analogous way. Obviously there's a lot more to a modern general purpose OS than a simple network device, so I'm not sure, maybe it wouldn't work very well?

Re:Programmer's critique of missing structure of operating s... by dragonwriter on Wednesday February 19, 2020 @08:41AM (#22365306)
> Yes, text configuration files are dumb because they require N parser/script editors times M configurable programs.

No, they don't. It's N editors + M parsers (where M=programs) I suppose if you want really smart editors that are also developed completely independently, such that each require it's own parser, its N+M parsers plus N editors.

> Furthermore, what people really want is universal progmatic and CLI access to configuration

I'm fairly certain most people don't want anything CLI anything.

Re:Programmer's critique of missing structure of operating s... by imtringued on Wednesday February 19, 2020 @11:00AM (#22366649)
I want better CLIs. The author's criticism is spot on when it comes to those. The CLI flag format should be specific to the shell not specific to the program.
Re:Programmer's critique of missing structure of operating s... by bregma on Wednesday February 19, 2020 @08:54AM (#22365398)
> I'm fairly certain most people don't want anything CLI anything.

I'm fairly certain they all want CLI everything, it's just they don't know it yet.

Re:Programmer's critique of missing structure of operating s... by bvrmn on Wednesday February 19, 2020 @10:34AM (#22366356)
It's more convenient to call and export needed function directly via C API than force everybody to make and use another "universal" layer.
Re:Programmer's critique of missing structure of operating s... by zdw on Wednesday February 19, 2020 @08:55AM (#22365406)
The low hanging fruit is a common parser/serializer, which is usually handled in library code in Unix - handwritten parsers are usually the enemy.

The other bits describe a whole configuration management system, which gets into the realm of questions like "For version X of software Y, value Z can be between A and B, but this changes when using version X+1", which is extremely difficult logic to encode into an external data store. Even worse are relational consistency issues that come up across structures.

Having a "fail if invalid" results in people setting values and then being frustrated at the system.

BTW, this all existed in a language neutral way in the early 2000's with XML parsing and validation leveraging RelaxNG and Schematron. Unfortunately those were deemed ugly and hard to use, and thrown out in exchange for the half baked JSON/YAML solutions.

Re:Programmer's critique of missing structure of operating s... by gen220 on Wednesday February 19, 2020 @09:05AM (#22365469)
I know it’s a bit hacky, but you can satisfy most of your requirements with environment variables; and most of your other requirements with a mini daemon that sits on top of `env`.

When I think about it, this is basically what the various “keychain” daemons provide, which IMO are underused despite their terrible ergonomics.

Re:Programmer's critique of missing structure of operating s... by laughingbovine on Wednesday February 19, 2020 @02:27PM (#22369079)
This sounds like your basic configuration management tool plus a service discovery tool.
Re:Programmer's critique of missing structure of operating s... by epr on Wednesday February 19, 2020 @10:27AM (#22366264)
> So now you turn one problem into two problems: manage the schemas and manage the data

This seems to completely ignore the fact that we are already managing schemas right now in the form of ad-hoc parsers and serializers which are arguably much worse than a more formally specified alternative.

Re:Programmer's critique of missing structure of operating s... by pjc50 on Wednesday February 19, 2020 @10:34AM (#22366351)
The HTML vs XHTML situation: would you prefer a hard failure on any error, or a graceful degradation system that is therefore always slightly degraded?

We go back and forth on this because tightening the schema only works when you can adequately define the requirements of both ends up front, and there isn't a vendor battleground happening in your standard. Developers end up escaping into an unstructured-but-working zone. Classics like "it's such a hassle to get the DBA to add columns, so we'll add a single text column and keep all the data in there as JSON".

Re:Programmer's critique of missing structure of operating s... by epr on Wednesday February 19, 2020 @11:26AM (#22366979)
> tightening the schema only works when you can adequately define the requirements of both ends up front

There are degrees of tightening though, aren't there? It would be a matter of necessity (to me, at least) that an interface like he describes be extensible. To me, it's about better OS primitives with more guarantees, etc. Versioned APIs with options to fallback to older versions (to at least some extent) seems like an obvious outcome. The last thing anyone would want would be the equivalent of JSON in a database column, which would completely defeat the purpose.

Re:Programmer's critique of missing structure of operating s... by pdimitar on Saturday February 22, 2020 @08:31AM (#22390887)
> The HTML vs XHTML situation: would you prefer a hard failure on any error, or a graceful degradation system that is therefore always slightly degraded?

Definitely would prefer the hard failure. The community will literally fix all of that in a week -- all template engines and DOM generators will adapt to avoid the hard errors and everything has a good shot at becoming better as a result.

Re:Programmer's critique of missing structure of operating s... by tyingq on Wednesday February 19, 2020 @08:21AM (#22365169)
"The database for a filesystem is an classic. WinFS, the filesystem that should have been a key feature of Longhorn/Windows Vista is based on a relational database."

I always wondered why Unix never added record-based files, in addition to stream-based files...like mainframes have. That would have simplified many things.

Re:Programmer's critique of missing structure of operating s... by bregma on Wednesday February 19, 2020 @08:57AM (#22365425)
> I always wondered why Unix never added record-based files, in addition to stream-based files...like mainframes have. That would have simplified many things.

You can implement record-based files (RAX, ISAM, etc) on top of stream-based files. The Unix philosophy was like Lego: small parts that do simple things that can be put together in combinations to build greater things. If you can build it from the basic blocks, it's not a basic block.

There are plenty of more advanced storage systems available for Unix. You don't need the OS vendor to supply The One.

Re:Programmer's critique of missing structure of operating s... by imtringued on Wednesday February 19, 2020 @11:05AM (#22366716)
>If you can build it from the basic blocks, it's not a basic block.

The idea behind the author's post is that a directed acyclic graph is a basic block that is just one layer of abstraction above raw bytes. JSON, YAML, TOML, INI, .properties, environment variables are all just different ways to represent a graph and if it were possible to store the graph directly in the filesystem then one would not have to manually [0] serialize the graph to bytes and deserialize it back later.

[0] graph -> byte conversion is the job of the graph filesystem

Re:Programmer's critique of missing structure of operating s... by Bystroushaak on Wednesday February 19, 2020 @01:15PM (#22368292)
Thank you for pointing that out, thats exactly one of my points.
Re:Programmer's critique of missing structure of operating s... by zaarn on Wednesday February 19, 2020 @09:17AM (#22365568)
The problem is triviality to implement, a similar problem occurs with TCP and UDP.

Lots of people implement half of TCP over UDP because they want packet based transmission but also care about losing packets.

Implementing packets on top of a stream is not as trivial as implementing a stream over packets.

Implementing streaming files over record-based files is easier, you just iterate over each record and append to the stream. Implementing records in stream-based files means you have to define a framing format, internal format, bookkeeping, etc.

Record-based files would be the proper lego building blocks.

Re:Programmer's critique of missing structure of operating s... by GoblinSlayer on Wednesday February 19, 2020 @10:08AM (#22366066)
The disk is a block device, i.e. it reads and writes data in blocks, the system just abstracts that into a stream interface.
Re:Programmer's critique of missing structure of operating s... by zaarn on Wednesday February 19, 2020 @11:24AM (#22366953)
Yeah and that is IMO the wrong abstraction. Blocks on disk should be organized into records, where multiple blocks can belong to a single or multiple (if using dedup or Copy-on-Write) records. Implementing a performant stream over that can be done in libc with minimal overhead.
Re:Programmer's critique of missing structure of operating s... by GoblinSlayer on Thursday February 20, 2020 @03:33AM (#22373612)
Now you want a different block structure.
Re:Programmer's critique of missing structure of operating s... by zaarn on Thursday February 20, 2020 @05:25AM (#22374052)
No? ZFS already organizes blocks on disks this way to some extend and Btrfs uses extends which include multiple blocks together and can be referenced multiple times.
Re:Programmer's critique of missing structure of operating s... by imtringued on Wednesday February 19, 2020 @11:13AM (#22366812)
The problem with the TCP vs UDP debate is that there exists a middle ground that satisfies most people's requirements.

TCP offers a single reliable globally ordered stream.

UDP offers unreliable packets in arbitrary order.

What people usually want is to send whole packets(or multiple streams) in arbitrary order with configurable reliability.

TCP and UDP offer neither but UDP has the least restrictions which makes it the only protocol on which you can implement your custom protocol.

Re:Programmer's critique of missing structure of operating s... by bvrmn on Wednesday February 19, 2020 @10:38AM (#22366396)
> Implementing streaming files over record-based files is easier

What a performance-wise nightmare. Goodbye page cache and other zero copy optimizations.

Re:Programmer's critique of missing structure of operating s... by zaarn on Wednesday February 19, 2020 @11:22AM (#22366936)
Not really, the OS still has to abstract it onto the harddisk, this should be trivial to achieve without impacting page cache and zero copy optimizations if the on-disk metadata is seperated sufficiently. COW Filesystems already do it to a lesser degree and they're certainly fine.
Re:Programmer's critique of missing structure of operating s... by bvrmn on Wednesday February 19, 2020 @11:41AM (#22367167)
What to do with clients who want a stream of records? Separated meta + payload allows fast byte streams. Combined meta + payload allows fast record streams. Structured implementation can't provide performance for both cases. But byte-oriented can. And we see this approach in modern and fast OSes.
Re:Programmer's critique of missing structure of operating s... by zaarn on Wednesday February 19, 2020 @11:43AM (#22367197)
Allow clients to choose which they receive?
Re:Programmer's critique of missing structure of operating s... by bvrmn on Wednesday February 19, 2020 @11:56AM (#22367351)
May be I was unclear. If we introduce record abstraction for file ops then OS low-level implementation have to decide which way to store structured data. meta + payload or meta separated from payload. In latter case there is no way to provide fast record streams. Former case does not allow fast byte streams. Client decision to receive bytes or records doesn't magically zero copy reorder stored data.
Re:Programmer's critique of missing structure of operating s... by zaarn on Wednesday February 19, 2020 @11:59AM (#22367398)
Why would the later cause record streams to be slow? Snapshots in COW FS' aren't that terribly slow either, even if there is quite a few of them, which basically boils down to the same structure, really.
Re:Programmer's critique of missing structure of operating s... by tyingq on Wednesday February 19, 2020 @10:07AM (#22366061)
"You can implement record-based files (RAX, ISAM, etc) on top of stream-based files"

Sure. And we have, many times over. A base implementation that supported fixed and variable records, shared process mmap() access, etc...would have value. And perhaps the kernel being aware of it would have some benefit.

Re:Programmer's critique of missing structure of operating s... by pdimitar on Saturday February 22, 2020 @08:40AM (#22390927)
> The Unix philosophy was like Lego: small parts that do simple things that can be put together in combinations to build greater things. If you can build it from the basic blocks, it's not a basic block.

And the normal programmer's logic is: "if it's not already integrated in the OS then I am not going to use it". So stream files it is, forever.

I think history has proven this. People are very risk-averse even if the risk is quite minor (example: using sqlite3 instead of the file system).

Re:Programmer's critique of missing structure of operating s... by ajross on Wednesday February 19, 2020 @09:41AM (#22365776)
Exactly. And with more opinion behind it: the fact that most of those items were shipped in a extraordinarily well-supported, mass market OS literally decades ago and still didn't catch on maybe says something about the value of the design ideas.
Re:Programmer's critique of missing structure of operating s... by jamesrcole on Thursday February 20, 2020 @03:14AM (#22373535)
> the fact that most of those items were shipped in a extraordinarily well-supported, mass market OS literally decades ago and still didn't catch on maybe says something about the value of the design ideas.

that statement seems heavily draw on the idea that success is the result of a meritocracy. But 'how good something is' often plays a small part in the selection of winners and losers.

Re:Programmer's critique of missing structure of operating s... by ajross on Thursday February 20, 2020 @11:26AM (#22376183)
In software architecture, it's really the only good evidence we have, though. I'm all for better metrics, but there aren't any. The point is just that none of these ideas are remotely "new", they've been heavily pushed by one of the biggest players in the industry for decades, and if anything was really beneficial in a profound way we'd surely know about it by now.

But we don't. So, no, that doesn't prove they're bad ideas. It does show that whatever benefit they have is underneath the noise floor of the experiment, though.

Re:Programmer's critique of missing structure of operating s... by sl1ck731 on Wednesday February 19, 2020 @10:07AM (#22366051)
I think the only thing in the parent's list that didn't catch on was WinFS, which I believe is because it never even shipped if I'm not mistaken.

Registry and Powershell are integral parts of being a Windows administrator.

I might be leading a little, but just because nix didn't choose to use these things doesn't imply they are bad. Just nix is "different".

Re:Programmer's critique of missing structure of operating s... by ajross on Wednesday February 19, 2020 @11:46AM (#22367220)
Right, I meant "catch on" in the sense of being clearly superior to the point that everything feels they need to have an answer for it and move to it. In fact the Registry[1] and Powershell are uniformly considered controversial even within their problem domain.

And I'm not arguing these things are "bad", exactly. I'm saying they aren't the panacea claimed in the article, that frankly they provide limited real benefit in a practical sense, and that the proof of this is that the one system that tried to implement them generically is aging glacially into legacy status without these ideas having propagated.

Basically: if these things were that great, at some point other systems would have picked them up while they moved on past windows. But they didn't. So... I guess my point is "why should we care about this stuff specifically?".

[1] Obviously the registry was emulated in the Linux world with the gconf/dconf stuff, which likewise was quite controversial and hasn't had anything close to universal adoption.

Re:Programmer's critique of missing structure of operating s... by laumars on Wednesday February 19, 2020 @06:33AM (#22364686)
File system allocation tables effectively are highly optimised databases already. I think the issue isn't the file system itself but rather the OS syscalls.
Re:Programmer's critique of missing structure of operating s... by naasking on Wednesday February 19, 2020 @08:48AM (#22365355)
> File system allocation tables effectively are highly optimised databases already. I think the issue isn't the file system itself but rather the OS syscalls.

It really is the file systems as well. If allocation tables are highly optimized databases, they're not the kind of safe and robust database we're used to:

https://danluu.com/file-consistency/

Re:Programmer's critique of missing structure of operating s... by laumars on Thursday February 20, 2020 @12:04PM (#22376513)
> If allocation tables are highly optimized databases, they're not the kind of safe and robust database we're used to

Those two points aren't mutually exclusive (that link, by the way, is discussing syscalls like I'd mentioned).

When I talk about "optimised" I mean in terms of other limits placed on the table that makes sense with regards to storing file metadata but wouldn't be desirable for a RDBMS.