News for hackers. Stuff that matters.
Re:Unix Commands I Wish I’d Discovered Years Earlier by ck2 on Tuesday September 10, 2013 @01:06PM (#6361356)
This is a nasty hack where it is counting the output from find.

What do you do when find cannot handle the sheer number of files, argument too long from the linux hardcoded MAX_ARG_PAGES

We need a dedicated utility that counts inodes.

Re:Unix Commands I Wish I’d Discovered Years Earlier by js2 on Tuesday September 10, 2013 @01:39PM (#6361562)
That is not at all correct.

The output from find is a list of filenames separated by newlines. The wc command is reading from stdin and counting those newlines[1] -- wc itself has just two arguments in its argv array here ("wc", "-l"). You will not exceed its argument list. Further, it is not at all a hack to count the number of files this way; rather using pipes to compose functionality is the Unix way[2].

But fwiw, if you want a count of the inodes in use in the entire filesystem, you can get that directly from "df -i".

In any case, as an exercise to the reader, I encourage you to grab the source to find and add the -count output option you desire.

[1] Technically a filename can contain newlines, so this would throw off the count. The fix for that would be to use find's -print0 output option and then pipe to something which can count on the nulls. In practice, you're unlikely to have such filenames.

[2] cf. http://harmful.cat-v.org/cat-v/unix_prog_design.pdf

Re:Unix Commands I Wish I’d Discovered Years Earlier by teddyh on Tuesday September 10, 2013 @01:33PM (#6361531)
I think you are confused — at no time are any files passed as arguments in the command "find . | wc -l".

Also, to count the number of files in a directory, you need to do readdir(), which will get you the names. Then, to see if any of them are directories to be recursed into you need to stat() the names individually. This is all "find" does. How, I'd very much like to know, is this "a nasty hack"?

Re:Unix Commands I Wish I’d Discovered Years Earlier by ck2 on Tuesday September 10, 2013 @03:55PM (#6362935)
I am not saying find is outputting files, I am saying it is decoding the full table entry for each file and formatting it, then passing that listing as an entry to wc. Which has to be a bunch of overhead that adds up when you have hundreds of thousands of files to count.

All we want from find is "count the number of inode entries in this tree branch that are type f". No formatting, no output, just count. That has to be much more efficient.

Re:Unix Commands I Wish I’d Discovered Years Earlier by wiredfool on Tuesday September 10, 2013 @04:24PM (#6363192)
OMM, looking at 700k files on a spinning disk, I'm getting an uncached run at 1m13s, and a cached run at 1.2 seconds.

That's "time find . | wc -l"

It's going to be disk limited unless you're in cache.

Switching to "time find . -printf '1' | wc -c" lowers overhead to .7 seconds, so roughly 1us/file.

Re:Unix Commands I Wish I’d Discovered Years Earlier by jbarber on Wednesday September 11, 2013 @04:34AM (#6366132)
There is a more important benefit to using the -prinf '1' approach - which is that it won't miscount the result if any of the filenames contain newlines.
Re:Unix Commands I Wish I’d Discovered Years Earlier by res0nat0r on Tuesday September 10, 2013 @04:05PM (#6363022)
You are going to need to rely on the filesystem authors to write a tool to traverse their inode trees directly, or probably write something like that yourself.

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux....

Re:Unix Commands I Wish I’d Discovered Years Earlier by clarry on Tuesday September 10, 2013 @01:32PM (#6361520)
In this invocation find and wc both have two fixed arguments, you're never going to hit MAX_ARG_PAGES. The output from find is passed to wc over a pipe, and to my knowledge there's no upper limit on how much data you can transfer over a pipe. I've piped huge disk images.
Re:Unix Commands I Wish I’d Discovered Years Earlier by inyourtenement on Tuesday September 10, 2013 @01:34PM (#6361542)
The only argument to his find command is '.', so where is the concern about argument list?