News for hackers. Stuff that matters.
Re:Freqfs: In-memory filesystem cache for Rust by weinzierl on Friday October 01, 2021 @04:18PM (#28722988)
We've been there. Before we had filesystems as we know them today, there were many different ways of persistent data storage. Roughly these could be grouped into two camps: The files camp and the records camp.

The record based approach had many properties we know from modern databases. It was a first class citizen on the mainframe and IBM was its champion.

In my opinion hierarchical filesystems won as everyday data storage because of their simplicity and not despite it. I think the idea of a file being just a series of bytes and leaving the interpretation to the application is ingenious. That doesn't mean there is no room for standardized OS-level database-like storage. In fact I'd love to see that.

Re:Freqfs: In-memory filesystem cache for Rust by rkagerer on Friday October 01, 2021 @09:50PM (#28725296)
I've always been annoyed how searching a few hundred thousand NTFS records for a filename containing some arbitrary text takes a relatively long time - even using specialized tools like FileLocator Pro which I believe directly scan low-level structures like the MFT - while I can do an equivalent search in a SQL database in milliseconds. I wish filesystems like that one had vastly more performant indexing structures for the metadata (without relying on add-on layers that defer indexing and - at least in my experience - tend to break down or be out of date or obfuscate files they think you don't care about - I'm looking at you, Windows Search).
Re:Freqfs: In-memory filesystem cache for Rust by RicardoLuis0 on Friday October 01, 2021 @11:39PM (#28725746)
I've been using a free tool by voidtools tool called "Everything" for years, it provides almost-instant search on windows NTFS volumes: https://www.voidtools.com/support/everything/
Re:Freqfs: In-memory filesystem cache for Rust by rkagerer on Saturday October 02, 2021 @01:23AM (#28726187)
Thanks. I used it years ago but it didn't suit my needs. I haven't found an indexing tool that does.

It's the difference between synchronous indexing that's baked into the system (as in file system metadata structures and database indexes, which update at the same time your data is changed) vs. fragile add-ons that index asynchronously (which in general I find tend to be too slow to update, missing results, and prone to breaking).

Re:Freqfs: In-memory filesystem cache for Rust by solidsnack9000 on Saturday October 02, 2021 @01:26AM (#28726200)
The lack of structure is ultimately why writes to filesystems are comparatively fast and reliable, though.
Re:Freqfs: In-memory filesystem cache for Rust by eximius on Saturday October 02, 2021 @12:13AM (#28725899)
well, there is a good reason they cant do that. they dont know what the files are.

perhaps filesystems should be extensible in a way that supports indexing intelligently.

(i know you mentioned an aversion to addons)

Re:Freqfs: In-memory filesystem cache for Rust by Koshkin on Friday October 01, 2021 @04:30PM (#28723096)
Modern filesystems in a way combine both approaches - they store the data unstructured but give the ability to also store metadata (attributes) in a structured way.
Re:Freqfs: In-memory filesystem cache for Rust by refenestrator on Friday October 01, 2021 @07:10PM (#28724425)
That functionality is mostly an afterthought in ext and ntfs, though. More of a big deal for Apple with hfs but still not something you'd build a database on.
Re:Freqfs: In-memory filesystem cache for Rust by rektide on Friday October 01, 2021 @11:21PM (#28725659)
> I think the idea of a file being just a series of bytes and leaving the interpretation to the application is ingenious.

the file as an opaque box for applications to store a real data structure is poisonously anti-file. it's totally what files are, what we think of them, but imo, systems like 9p, or linux's procfs or sysfs are The True Way for files: small discrete pieces of data which are part of a system of directories tlthat express a larger compilated hierarchical system of data.

Files won, but only the stupidest wrongest version. Easy to copy and manage but utterly useless on their own, unscriptable, pointless eithout their complex applications there to use them.

I dont think db's/records are that interesting either. i think we just need to really try files. Fine grained files. As opposed to these big ole blobs the OS cant really interact with.

Re:Freqfs: In-memory filesystem cache for Rust by remexre on Friday October 01, 2021 @11:39PM (#28725749)
Don't you need mandatory locking of files and directories, or rather powerful transactional semantics for the filesystem then?
Re:Freqfs: In-memory filesystem cache for Rust by rektide on Saturday October 02, 2021 @12:39PM (#28729998)
A lot of filesystems have snapshots. NTFS & others have transactional capabilities. I don't regard locking as necessary or helpful when the OS can provide these capabilities.
Re:Freqfs: In-memory filesystem cache for Rust by mike_hock on Saturday October 02, 2021 @07:13AM (#28727664)
There aren't really any hurdles to implementing an SQL database on top of a plain block device, are there? So I wonder why no-one has gone there. This would allow the database server to do caching in a way that makes sense for the database and not have to hope that the filesystem cache does the right thing.
Re:Freqfs: In-memory filesystem cache for Rust by gpderetta on Saturday October 02, 2021 @09:12AM (#28728294)
Commercial databases routinely do exactly that. O_DIRECT basically exists because Oracle needed it