News for hackers. Stuff that matters.
Re:Sqlitefs: SQLite as a Filesystem by szundi on Saturday July 27, 2024 @08:05AM (#41086028)
Filesystems are DB like structures these days anyway.
Re:Sqlitefs: SQLite as a Filesystem by jeltz on Saturday July 27, 2024 @08:23AM (#41086117)
Are they? From my experience from databases and some limited from filesystems I would disagree. Sure, there are some similarities but not that much.
Re:Sqlitefs: SQLite as a Filesystem by pdimitar on Saturday July 27, 2024 @08:30AM (#41086158)
Where do you get that from? In my practice this was never true and still is not.
Re:Sqlitefs: SQLite as a Filesystem by chipdart on Saturday July 27, 2024 @08:53AM (#41086259)
> Filesystems are DB like structures these days anyway.

I think you're confusing the fact that an application uses data structures with that application being like a database.

Re:Sqlitefs: SQLite as a Filesystem by lupire on Saturday July 27, 2024 @09:23AM (#41086434)
Other way around. Applications use databases instead of the filesystem.
Re:Sqlitefs: SQLite as a Filesystem by tom1337 on Saturday July 27, 2024 @08:27AM (#41086146)
Ok so if we take this repo and then the headline from https://news.ycombinator.com/item?id=41085376, we could get an infinitely fast file system by just putting another SQLite filesystem on top.
Re:Sqlitefs: SQLite as a Filesystem by bee_rider on Saturday July 27, 2024 @09:15AM (#41086378)
IMO in this fantasy it should go like (1-.35)^n, not 1-.35n. We must keep our harebrained schemes plausible.
Re:Sqlitefs: SQLite as a Filesystem by bhaney on Saturday July 27, 2024 @09:03AM (#41086312)
There's a recursive CTE joke in here somewhere
Re:Sqlitefs: SQLite as a Filesystem by bilekas on Saturday July 27, 2024 @09:10AM (#41086353)
I was just thinking the same thing.. Im supposed to be on holidays but damn it I need to test this.
Re:Sqlitefs: SQLite as a Filesystem by punnerud on Saturday July 27, 2024 @08:28AM (#41086150)
If SQLite is faster than a normal file system, how fast is SQLite running on “SQLite as a filesystem”?
Re:Sqlitefs: SQLite as a Filesystem by zoky on Saturday July 27, 2024 @09:04AM (#41086319)
It turns out if you run SQLiteFS on SQLiteFS, performance increases exponentially, and as the number of SQLiteFS layers increases, seek and read times asymptotically approach zero. In other words, this is the world’s first O(0) file system.

At this rate, with only a few performance tweaks, SQLiteFS should be able to retrieve files before you even ask for them.

Re:Sqlitefs: SQLite as a Filesystem by smitty1e on Saturday July 27, 2024 @12:17PM (#41087497)
Less humorously, if one were calculating a few files for compilation, or caching for a service, a SqliteFS using :memory: might be just the thing.
Re:Sqlitefs: SQLite as a Filesystem by earthboundkid on Saturday July 27, 2024 @09:20AM (#41086411)
The problem is that it sends you files before you need them, which clogs up throughput as you try to figure out what the files are supposed to be for. It's more efficient to slow everything back down to NVMe speeds.
Re:Sqlitefs: SQLite as a Filesystem by bilekas on Saturday July 27, 2024 @09:12AM (#41086361)
Jokes aside, if I understand correctly it's still assumed the performance increase from the sqlLite is down to the reduced number of OPEN and CLOSE calls. So if that's the case, then it will be as slow as however many sqllites you're running? Meaning it should in theory lose performance?!
Re:Sqlitefs: SQLite as a Filesystem by lupire on Saturday July 27, 2024 @09:22AM (#41086427)
Obviously adding redundant layers will reduce performance.
Re:Sqlitefs: SQLite as a Filesystem by jefebromden on Saturday July 27, 2024 @08:34AM (#41086173)
Will anyone mention the repo hasn't been updated on 4 years?
Re:Sqlitefs: SQLite as a Filesystem by willcipriano on Saturday July 27, 2024 @09:10AM (#41086351)
When have you last updated your filesystem?
Re:Sqlitefs: SQLite as a Filesystem by endorphine on Saturday July 27, 2024 @09:17AM (#41086394)
A fair analogy would be: "when was upstream code of the filesystem was last updated?".

To that question, I doubt one would find a polular filesystem that was updated longer than 1 year ago (maybe I'm too liberal).

Re:Sqlitefs: SQLite as a Filesystem by Jach on Saturday July 27, 2024 @09:32AM (#41086484)

    # zpool history | grep upgrade
Looks like last May.
Re:Sqlitefs: SQLite as a Filesystem by yjftsjthsd-h on Saturday July 27, 2024 @10:14AM (#41086734)
That's the on-disk format, you probably updated the actual driver code a lot more recently than that
Re:Sqlitefs: SQLite as a Filesystem by Jach on Saturday July 27, 2024 @11:39AM (#41087249)
They didn't specify what kind of update, but in this case no, I'm on zfs-2.2.3, which was released in February. I'd have to go check out when it was marked stable by Gentoo and I actually updated. (I do occasionally have to rebuild whatever version I'm on when upgrading kernel versions.)
Re:Sqlitefs: SQLite as a Filesystem by willcipriano on Saturday July 27, 2024 @11:29AM (#41087193)
What was in that update that you are worried about this missing?
Re:Sqlitefs: SQLite as a Filesystem by Jach on Saturday July 27, 2024 @11:40AM (#41087250)
Re:Sqlitefs: SQLite as a Filesystem by kachapopopow on Saturday July 27, 2024 @09:58AM (#41086638)
yesterday, zfs.
Re:Sqlitefs: SQLite as a Filesystem by alberth on Saturday July 27, 2024 @08:47AM (#41086229)
Reminds me of Microsoft WinFS.

Which was essentially SQL Server as a filesystem.

Project was ultimately cancelled.

https://en.m.wikipedia.org/wiki/WinFS

Re:Sqlitefs: SQLite as a Filesystem by abofh on Saturday July 27, 2024 @10:37AM (#41086880)
I think I interviewed with that team (or tried to at least) - it had great concepts, but the limited leadership I met on that one seemed unlikely to actually get to product (I say with 20 years retrospection) - they were stuck on sql when it needed a directory, but the directory couldn't scale to handle filesystems. Both have since been fixed, but I think the idea is well dead in MS
Re:Sqlitefs: SQLite as a Filesystem by krylon on Saturday July 27, 2024 @09:27AM (#41086454)
I'm not a big fan of Windows, but I'm a bit sad that the project didn't go anywhere forever and then was cancelled. They had some pretty ambitious and interesting ideas for it.

E.g. the only public beta of Vista that had it enabled used the database as a backing store for Outlook contacts. Each contact was an object in the database. So were photos. And you could link - as I recall, I only read about it back then - people to photos. And then search on that data, of course. Like, "give me all emails, documents, and photos that have to do with person X, from 2022-2023".

But IIRC, the performance was atrocious. They might have been able to fix that, but instead it ended on the junk heap of unrealized ideas.

Re:Sqlitefs: SQLite as a Filesystem by HeckFeck on Saturday July 27, 2024 @09:44AM (#41086552)
I remember those Longhorn betas - yes, it was coming apart under feature creep, but that only gave us more to play with and anticipate in the leaks.

A shame that MS is no longer in the mindset of ambition like this. Instead we get adverts and clickbait in the Start Menu, and the worst filesystem search on any OS I've experienced.

Re:Sqlitefs: SQLite as a Filesystem by naikrovek on Saturday July 27, 2024 @01:16PM (#41087970)
I feel like I need to inform you that the people doing the “recommendations” in the Windows start menu are not the same people who write filesystems.

The venn diagram of those two sets of people is two non-touching, non-intersecting circles.

There are interesting things going on, I’m sure, but we won’t see any of it until this AI baloney is behind us.

Nerds create a virtual moron that gets a large percentage of stuff wrong and execs demand that it be inserted into every product made immediately. An absolute genius move only an MBA can explain.

Re:Sqlitefs: SQLite as a Filesystem by HeckFeck on Saturday July 27, 2024 @02:30PM (#41088408)
Sure, but that indicates institutional rot. If the people who write NTFS drivers (i.e. its devs who love Windows the most) can't push back on user hostility and short-sighted decisions are made constantly then we have a company that has drifted far from its strengths. It is just coasting on brand recognition. This is the kind of trend that is seldom reversed.
Re:Sqlitefs: SQLite as a Filesystem by greenthrow on Saturday July 27, 2024 @08:52AM (#41086254)
Why would you bother with this instead of just making an in memory drive? I remember mounting in memory drives even in MS-DOS back in the early 90s. I did it to move whatever game I wanted to play into the RAM Drive so my load times would be faster.
Re:Sqlitefs: SQLite as a Filesystem by BrandoElFollito on Saturday July 27, 2024 @09:14AM (#41086373)
I use this today on my Raspberry Pis to load the logs and other active IO to memory and avoid wearing out the SD card.
Re:Sqlitefs: SQLite as a Filesystem by greenthrow on Saturday July 27, 2024 @09:36AM (#41086519)
Ok but why not just use a memory drive directly? What is gained by adding SQLite as a layer?
Re:Sqlitefs: SQLite as a Filesystem by BrandoElFollito on Saturday July 27, 2024 @10:19AM (#41086770)
I am commenting on your "I remember mounting in memory drives even in MS-DOS back in the early 90s.". Just to say this this is still a thing today.

I also mounted drives in memory in early 90's :)

Re:Sqlitefs: SQLite as a Filesystem by bee_rider on Saturday July 27, 2024 @09:17AM (#41086392)
Videogames have gotten so ridiculously huge nowadays, we’ll be at the prosumer/workstation cusp to make this work nowadays.
Re:Sqlitefs: SQLite as a Filesystem by gbonc2 on Saturday July 27, 2024 @09:31AM (#41086474)
I wrote this one, I tested with PostgresDB and Oracle , at time:

https://github.com/gbonacini/dbfs

Re:Sqlitefs: SQLite as a Filesystem by adius on Saturday July 27, 2024 @12:41PM (#41087699)
Related: SQLiteDAV - WebDAV server that maps an SQLite database to directories/files.

https://github.com/Airsequel/SQLiteDAV

Re:Sqlitefs: SQLite as a Filesystem by DemocracyFTW2 on Saturday July 27, 2024 @06:34PM (#41089810)
It works (on Linux)! Should add that I only got to do what it does after following the advice expressed in the error message:

    fusermount: option allow_other only allowed if 'user_allow_other' is set in /etc/fuse.conf
It did not work using `sudo`
Re:Sqlitefs: SQLite as a Filesystem by sillywalk on Saturday July 27, 2024 @07:05PM (#41089991)
For the bored, I was searching for 'database filesystem' and found one by Oracle. I gather it's a way to make editing BLOBs easier. That could make for some interesting recursive things.

It's "a file system interface placed on top of files and directories that are stored in database tables."[0]

[0] https://docs.oracle.com/en/database/oracle/oracle-database/1...