News for hackers. Stuff that matters.
Re:You can list a directory containing 8 million files But n... by moe on Monday August 15, 2011 @06:55PM (#2889056)
Actually 'find' will also stat each entry no matter what.

Many of the standard-tools that most people would intuitively expect to be rather optimized (find, rsync, gzip) are embarrassingly inefficient under the hood and turn belly up when confronted with data of any significant size.

That probably stems from the fact that most of the development on these tools took place during a time when 1GB harddrives were "huge" and SMP was "high end".

Re:You can list a directory containing 8 million files But n... by tedunangst on Monday August 15, 2011 @10:13PM (#2889532)
The only issue I'm aware of with gzip is actually in zlib, where it stored 32-bit byte counters, but those were strictly optional and it works fine with data that overflowed them. The zlib window size may be only 32k, but bzip2 doesn't do that much better with 900k and a better algorithm, so I wouldn't consider it embarrassingly inefficient.
Re:You can list a directory containing 8 million files But n... by moe on Tuesday August 16, 2011 @07:20AM (#2890711)
I was referring to the lack of SMP support in gzip (see http://www.zlib.net/pigz/).
Re:You can list a directory containing 8 million files But n... by Steve_Baker on Tuesday August 16, 2011 @04:09PM (#2892677)
How do you tell a file from a directory without stat()ing it? The d_type field is not portable. Since find and other tools like it need to recursively descend a directory tree, a stat() for each file to determine its type is unavoidable.
Re:You can list a directory containing 8 million files But n... by Ralith on Monday August 15, 2011 @10:04PM (#2889510)
But times have changed, and development isn't dead. Why haven't they been updated? The optimizations you imply are often straightforward and well-understood; not major undertakings to implement.
Re:You can list a directory containing 8 million files But n... by sixtofour on Monday August 15, 2011 @10:15PM (#2889542)
"But times have changed, and development isn't dead. Why haven't they been updated?"

Maybe because listing 8M files is not a common use case, and there just isn't the motivation to update otherwise perfectly working code. It's not an itchy problem.

Re:You can list a directory containing 8 million files But n... by Ralith on Tuesday August 16, 2011 @08:15PM (#2893446)
Compressing/decompressing large quantities of data is, at the very least.