News for hackers. Stuff that matters.
Re:One Billion Files: Scalability Limits in Linux File Syste... by masklinn on Wednesday April 06, 2016 @11:24AM (#11439368)
> mkdir -p "0/2/7/8/c"

That's at most 16 entries per directory which seems unnecessarily low. Taking 2 chars/level gives 96 entries per directory which seems better, that's what Git uses (3 is 4096 which may be a tad high). Yields a flatter but not too broad hierarchy.

Re:One Billion Files: Scalability Limits in Linux File Syste... by lazyant on Wednesday April 06, 2016 @01:19PM (#11440167)
I must be stupid; with 2 hex chars (dir structure like 0a/7b/...) wouldn't we have 16*16 = 256 (edit: can't do math) entries per directory, not 96?
Re:One Billion Files: Scalability Limits in Linux File Syste... by masklinn on Wednesday April 06, 2016 @01:44PM (#11440358)
256 actually, but I have no idea where that 96 comes from, it was probably the alcohol typing (or I was already thinking 4096 and I wrote part of that instead of the correct number)
Re:One Billion Files: Scalability Limits in Linux File Syste... by tyingq on Wednesday April 06, 2016 @02:03PM (#11440505)
Worth mentioning that's "directory entries per directory". The "data files per directory" isn't driven by the intermediate path...just the total number of "full paths".
Re:One Billion Files: Scalability Limits in Linux File Syste... by tyingq on Wednesday April 06, 2016 @11:38AM (#11439471)
That's a reasonable optimization, but wouldn't help much with runtime, since you aren't traversing any of it, but rather opening with full paths. I suppose it might make backups, or some sort of monitoring more efficient.
Re:One Billion Files: Scalability Limits in Linux File Syste... by masklinn on Wednesday April 06, 2016 @12:02PM (#11439657)
> That's a reasonable optimization, but wouldn't help much with runtime, since you aren't traversing any of it

Depends what data you're storing (or not storing) I guess. If you're storing all metadata anyway, you may even discard the original file name entirely and have the full path be composed of hash sections. But you may want to traverse the tree (because you're not storing much/any) data, and split things out because there's a number of APIs which don't deal well with humongous directories.

Re:One Billion Files: Scalability Limits in Linux File Syste... by tyingq on Wednesday April 06, 2016 @12:12PM (#11439724)
Right, but open("a/b/c/d/e/f/0/1/2/3") isn't slower than open("ab/cd/ef/01/23"). Flatter only matters if you are traversing the hierarchy, which wouldn't be a normal use case for this.
Re:One Billion Files: Scalability Limits in Linux File Syste... by loeg on Wednesday April 06, 2016 @12:20PM (#11439782)
Sure it is. You have to do twice as many directory lookups, even if they're cached.
Re:One Billion Files: Scalability Limits in Linux File Syste... by tyingq on Wednesday April 06, 2016 @01:00PM (#11440028)
The open() system call isn't doing recursive "directory lookups" (like readdir()) per se. It does have to traverse a linked list (dentries) to map the path+filename into an inode. But 6 deep into a linked list vs 3 deep is, I suspect, hardly measurable in terms of elapsed time. It's all hashed, and cached.