The filesystem, and the file itself, are perhaps the best abstractions ever designed in computing. A database sometimes uses the file API when actually wants to access the disk in a lower-level fashion, for which APIs are fragmented, non stable, non portable. So it's not a fault of the abstraction, but the fact that a different low-level abstraction to access the storage in a way databases need to do is missing from POSIX.

Not really...

They are one of the _first_ abstractions, and they are quite fundamental. That's why they all still exist in the way they are.

Just look at the name: "file". And "directory". Where do these terms come from? Right, from organizing things, mainly offices. And offices are organized like that because this is what is practical under the constraints of humans working with paper in a three dimensional space.

But looking at it semantically, almost no one really wants to only use files or directories. We mostly want to use semantical tags. And one kind of tag can be a hierarchical "directory/file" tag - fair enough. But that's only a small part of the picture, because we also want to tag our objects/entities by date, creator, department, security clearance level, how long they have to be archived and so on. Using just a filesystem does not provide this, so we have to build on top of that.

And you can see that this is changing. Look at phones nowadays. They don't use this anymore. Pictures are stored in collections/groups, can be tagged and sorted in various ways. They can be seen on a geographical map. And much more. No one has a directory/file structure in their head.

So the question is: does the OS still need to support that? Actually, maybe not. Maybe SSDs today can work much more efficient if we don't assume everything must be a file.

Just wait, we will get there.

Completely agreed. Every serious project I've been in has been striving to invent storage that's as close to their problem as possible. And I do the same in my hobby projects.

The file/directory abstraction survived for a long time but nowadays it's often a limitation more than a convenience.

I think Ceph has the right model. The only abstraction is physical disk to addressable binary/object storage. Everything else is implemented on top of that so you can do S3, filesystems, databases all close to the metal.

> There's a kind of arrogance that we fall into making decisions for others when we design APIs, library interfaces, etc where we know what we're exposing and users don't know what's good for them. Maybe that's right sometimes, i don't know?

Usually it's right, and at least you're putting thought into it. API design is not easy, hence why it gets fucked up so often.

I just went through explaining to someone why they shouldn't go pawing through /proc to find details on a running process. How there were commands like 'strace' and 'lsof', or even running a debugger that could better achieve their goals. Often people just aren't aware.

The option to bypass layers of abstraction should be there, but the option should be off by default, and come with plenty of warnings akin to "if you're removing the cylinder head, we assume you know what you're doing, and you assume all responsibility."

Other databases already bypass the kernel routinely; I'm reminded of Oracle which could use a partition or even whole disk directly. And yes, I know SQLite is not like other databases; that's why it gets tricky.

Honestly, the stuff they lay out is the kind of customer requests I dream of getting. I would gladly implement these, where they don't break backwards compatibility, and where they are off by default but can be turned on, in a fashion where it doesn't upset/surprise/break currently majority usage.

Any abstraction leaks. Good abstractions do not leak for common use cases and provide necessary access to the internals for uncommon use cases.

Which os is a good abstraction by this metric?

Yes that's a leading question.

I consider the libraries of the languages that allow the programmers to do everything (e.g. Python or JavaScript) are good at this: The library authors signal that these functions should not be called and these can be, but ultimately if you’re monkeypatching or something you can really do this. Obj-C is also a good language in doing this too…

Which metric? Leaky abstractions is a "Law", not a metric.

Are you looking for experience?

[ see Wikipedia for contextual meaning of law here, https://en.wikipedia.org/wiki/Law_(principle) ]

I think by "metric" he means your definition of a good abstraction, not the fact that all abstractions leak.

>There's a kind of arrogance that we fall into making decisions for others when we design APIs, library interfaces, etc where we know what we're exposing and users don't know what's good for them

This pops up a lot when libraries are designed with functions that are non-public. I only want to access the underlying function, not use your abstraction on top of it. You can't anticipate every possible use case of your library!

> You can't anticipate every possible use case of your library!

On the flipside, I consider a successful software tool is one that is used to do something undreamt of by its author.

One should not try to imagine every possible use to which their software will be put, especially when designing abstractions or interfaces. But these abstractions should be flexible enough that they can be adapted for any use.

There might be very good reasons that the underlying function is not public (I can think of private methods in OO where end users absolutely should not be calling them). To me, this seems like the public interface is lacking or perhaps too rigid, or maybe not well-understood (either by designer or user).

In any case APIs are hard.

This is a great point, that APIs often abstract over things that some people need finer grained control over. I work on an API that was too high level, and was recently overhauled in a backward incompatible way due to the limits inherent in the abstractions, limits and abstractions that most of the time are a good idea, but were getting in the way of professionals. So I’m inclined to agree on one hand.

On the other hand, the OS filesystem APIs are detailed enough for most people (and even for most of the OS internals too). I’ve never wanted to know sector sizes or power loss characteristics for any reason other than curiosity. The needs of SQLite aren’t very common. Most developers writing code to use the filesystem don’t need (or even want) to know if a file was renamed or what happened to it before last reboot or whether it can be mmap()ed with other processes. I can barely manage to use rsync, and if I was burdened with these things whenever I need to open or write a file, I’d use SQLite as my API instead of fopen(), which puts me right back at using a high level API (https://www.sqlite.org/whentouse.html).

All APIs present an interface over the resources they manage, so an abstraction is inherent, even if the API is low level. But usually part of the goal of API design is to make the interface easier to use than the resources it manages. If we expose an API that is too low level, then we compromise on usability and safety for everyone in favor of performance and detailed control for a very small minority of users.

So personally, I think I agree with your general concern in the sense that API designers should be aware of the needs of who they’re designing for, but I also think SQLite and the Linux filesystem aren’t a good example.

I’m not sure if I would call SQLite’s needs incommon — in a perfect world, every app should have SQLite’s properties like preserving user data on power loss, etc… It’s just that other apps just don’t care enough about these issues as much as SQLite.

For example, I’m pretty sure I would like every app to automatically handle cases where the underlying file gets renamed or removed. (AFAIK apps that use the Cocoa API automatically does these.) However there really isn’t any POSIX API that allows handling these kind of situations — and to do these now you need to do all of these nitty-gritty things that SQLite does.

In fact, I think that the reason why SQLite needs all of these nitty-gritty details that does everything is because of an insufficient abstraction. If the file APIs were better and didn’t require everyone to consider whether you have to call fsync or not, SQLite wouldn’t have needed any of these. But well POSIX have flied and here we are.

The API doesn’t need to expose whether data is preserved on power loss. Normally these days, it is, and it’s implemented underneath the API.

I don’t think the POSIX API is perfect by any stretch, but there is a real downside to going lower level, in terms of productivity, in terms of mis-use and accidents, in terms of security, and in terms of ability to support old hardware, heterogeneous storage devices, networked storage devices, etc. If there was a huge and unanimous demand for the low level features SQLite uses, the API would already expose those features.

APIs sometimes are too high level, and sometimes are too low level. Getting it wrong in either direction has negative consequences. And getting it exactly right is hard. In any case, I’m unconvinced that the Linux filesystem API is abstracted at the wrong level or represents a good example of an API that is too high level.

Files are a good abstraction to start from.

But to be honest, persistence & durability of data are the goal of Storage and thus what the filesystem API needs to be designed for.

The API needs to clearly be designed for atomicity, durability & to allow coordination across multi-file structures.

This can be achieved by either 1) indicating all required synchronizations to callers in metadata, or 2) fully encapsulating durability by making the API atomically commit every operation.

I don't agree these factors should be hidden, since a losing/ corrupting data are fundamentally at odds with the goals of a Storage API.

It's not just SQLite, any application needing to maintain a datastore needs reliable storage.

> What do you think? Have we all been missing something like this in api design?

No. It's hard to design, maybe harder to design an API and perhaps a misunderstanding when APIs are designed for others or an API just for an API.

As others have already commented about the leaking abstractions (The Law of Leaky Abstractions) which is a good description IMHO to make the non-boundary clear even when it was (non-)intended.

This can be found as well in Hyrum's Law, which IMHO is from a different point of view but reminds as well that the concept to just API for API is conceptually broken.

> Sqlite is all like, "yeah but we need to know what you're abstracting or everything is terrible."

I'm wondering where you pick the terrible from, I have not found that in the article. And it's absolutely not "all like". The paper describes common problems with file-systems on what Sqlite does which is standard programming - at least for Sqlite authors.

You seem not to be much aware of these kind of problems and with all due respect, this creates the impression that you have not experienced this much yet. This aligns well with the localization by yours as something "missing in the design" where I'd like to offer a different view: It's missing practice. To keep the colloquial tone:

Despite what has been missed in API design, we all haven been missing practising writing software.

https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a... https://www.hyrumslaw.com/

> There's a kind of arrogance that we fall into making decisions for others when we design APIs, library interfaces, etc where we know what we're exposing and users don't know what's good for them. Maybe that's right sometimes, i don't know?

So when someone who creates an Open-Source library creates an API that you don't like, they are arrogant? Interesting take.