A faster way to delete millions of files in a directory

Re:A faster way to delete millions of files in a directory by js2 on Friday May 31, 2013 @09:38PM (#5802229)

FWIW, a directory with millions of files is likely to be quite large (I'm referring to the directory inode itself, which contains a mapping of filenames to inodes). Depending upon the file system, reclaiming the space used by all those millions of mappings might require creating a new directory into which to move the remaining files.

BTW, having millions of files in an ext3 directory in the first place is probably a bad idea. Instead, layer the files into two or three directory levels. See here:

http://www.redhat.com/archives/ext3-users/2007-August/msg000...

(Git for example places its objects under 1 of 256 directories based on the first hex byte representation of the object's SHA-1.)

[ Parent | on HN ]

Re:A faster way to delete millions of files in a directory by protomyth on Saturday June 01, 2013 @12:22PM (#5804586)

As I remember it, HP-UX had some very poor performance characteristics once a single directory started to get into the thousands of files. It slow down for all read / write operations in that directory. We are talking multiple seconds to read a small 1K file much less write.

[ Parent | on HN ]

Re:A faster way to delete millions of files in a directory by justincormack on Saturday June 01, 2013 @05:48AM (#5803398)

This has got much better over time. A filesystem should be a database (for larger blobs of data) so it should work, but scalability is limited still. Newer fs may be better.

[ Parent | on HN ]