News for hackers. Stuff that matters.

A comment on One Billion Files: Scalability Limits in Linux File Systems (2010) [pdf]. Parent comment

Re:One Billion Files: Scalability Limits in Linux File Syste... by masklinn on Wednesday April 06, 2016 @12:02PM (#11439657)
> That's a reasonable optimization, but wouldn't help much with runtime, since you aren't traversing any of it

Depends what data you're storing (or not storing) I guess. If you're storing all metadata anyway, you may even discard the original file name entirely and have the full path be composed of hash sections. But you may want to traverse the tree (because you're not storing much/any) data, and split things out because there's a number of APIs which don't deal well with humongous directories.

[ Parent | on HN ]
Re:One Billion Files: Scalability Limits in Linux File Syste... by tyingq on Wednesday April 06, 2016 @12:12PM (#11439724)
Right, but open("a/b/c/d/e/f/0/1/2/3") isn't slower than open("ab/cd/ef/01/23"). Flatter only matters if you are traversing the hierarchy, which wouldn't be a normal use case for this.
[ Parent | on HN ]
Re:One Billion Files: Scalability Limits in Linux File Syste... by loeg on Wednesday April 06, 2016 @12:20PM (#11439782)
Sure it is. You have to do twice as many directory lookups, even if they're cached.
[ Parent | on HN ]
Re:One Billion Files: Scalability Limits in Linux File Syste... by tyingq on Wednesday April 06, 2016 @01:00PM (#11440028)
The open() system call isn't doing recursive "directory lookups" (like readdir()) per se. It does have to traverse a linked list (dentries) to map the path+filename into an inode. But 6 deep into a linked list vs 3 deep is, I suspect, hardly measurable in terms of elapsed time. It's all hashed, and cached.
[ Parent | on HN ]