News for hackers. Stuff that matters.
Re:Command-line tools can be faster than your Hadoop cluster by clarkm on Sunday January 18, 2015 @03:11PM (#8908668)
You can make it even faster by using fgrep since you're not searching for a regex.
Re:Command-line tools can be faster than your Hadoop cluster by simpleigh on Sunday January 18, 2015 @03:29PM (#8908736)
The article addresses this:

> The -F for grep indicates that we are only matching on fixed strings and not doing any fancy regex, and can offer a small speedup, which I did not notice in my testing.

I guess grep is probably clever enough to choose a faster matching algorithm once it's parsed the pattern and discovered it doesn't contain any regex fun.

Re:Command-line tools can be faster than your Hadoop cluster by clarkm on Sunday January 18, 2015 @05:57PM (#8909330)
In general grep seems smart enough that it would do that, but it hasn't been my experience. Just last week I was searching through a couple hundred gigs small xml files. I found that:

    $ LC_ALL=C fgrep -r STRING .
was much faster than plain grep. This was on a CentOS 5 box, so maybe newer versions of grep are smarter.

But then again, if I was on a newer box I'd just install and use ack or ag.

Re:Command-line tools can be faster than your Hadoop cluster by paralelogram on Sunday January 18, 2015 @06:52PM (#8909510)
LC_ALL=C makes grep faster because text matching is normally locale-sensitive, for example 'S' ('\x53' in Big5) is not a substring of '兄' ('\xA5\x53' in Big5).