News for hackers. Stuff that matters.
Re:SQLite the only database you will ever need in most cases by gugagore on Thursday April 15, 2021 @05:34AM (#26818218)
What about on the other end? Why not have each hex digit be a directory along a path? Then you have very, very few files per directory at the cost of deeper hierarchy. What's the practical downside?
Re:SQLite the only database you will ever need in most cases by pmiller2 on Thursday April 15, 2021 @07:36AM (#26819123)
Directories are just another type of file. So, if you do this, and your filenames are n characters long, you'll end up needing to do n file accesses just to find the file you're looking for. Unless the underlying file system does something to make that particular access pattern fast, well... it's going to be stupidly slow after a certain point.
Re:SQLite the only database you will ever need in most cases by loeg on Thursday April 15, 2021 @10:59AM (#26821269)
> Unless the underlying file system does something to make that particular access pattern fast

I don't know exactly how Linux does it. Windows hands off whole paths to the filesystem, so this idea is possible there.

In FreeBSD, there is a generic routine (lookup(9)) that goes component by component, so at each step the filesystem is only asked to resolve a single component to a vnode. I think a clever filesystem implementation (in FreeBSD) could look at the remaining path and kick off asynchronous prefetch... but I am not aware of anything doing this.

Re:SQLite the only database you will ever need in most cases by eru on Thursday April 15, 2021 @11:46PM (#26829207)
I've played around with fuse a bit.

There are two modes you can implement for your filesystem: in the so-called 'high level' mode you get the whole path. In the 'low level' mode the filesystem asks you for one piece of the path at a time.

The low level mode seemed faster in my tests, and I think it's also closer to how Linux kernel works internally?

Re:SQLite the only database you will ever need in most cases by ArneBab on Thursday April 15, 2021 @08:04AM (#26819379)
You then need to do more stat calls — one per hierarchy level — and you waste more inodes.
Re:SQLite the only database you will ever need in most cases by thewakalix on Thursday April 15, 2021 @06:37AM (#26818684)
[epistemic status: amateur speculation]

I’d expect it to take up slightly more disk space. More importantly, the more indirections, the longer it’ll take to get to the file.

Re:SQLite the only database you will ever need in most cases by loeg on Thursday April 15, 2021 @10:54AM (#26821217)
More overhead for each extra layer. Kernel serialization/locking, random seeks (dirent and inode per level), maybe other concerns.
Re:SQLite the only database you will ever need in most cases by eru on Thursday April 15, 2021 @11:48PM (#26829213)
The random seeks aren't a given: your filesystem could do something clever with memory or disk layout to avoid most of them.
Re:SQLite the only database you will ever need in most cases by loeg on Friday April 16, 2021 @10:27AM (#26834303)
Anything particular in mind? You could probably get a good academic paper or patent out of this.