Re:Command-line tools can be faster than your Hadoop cluster
by clarkm on Sunday January 18, 2015 @03:11PM
(#8908668)
You can make it even faster by using fgrep since you're not searching for a regex.
[ Parent
| on HN ]
Re:Command-line tools can be faster than your Hadoop cluster
by simpleigh on Sunday January 18, 2015 @03:29PM
(#8908736)
The article addresses this:
> The -F for grep indicates that we are only matching on fixed strings and not doing any fancy regex, and can offer a small speedup, which I did not notice in my testing.
I guess grep is probably clever enough to choose a faster matching algorithm once it's parsed the pattern and discovered it doesn't contain any regex fun.
Re:Command-line tools can be faster than your Hadoop cluster
by clarkm on Sunday January 18, 2015 @05:57PM
(#8909330)
In general grep seems smart enough that it would do that, but it hasn't been my experience. Just last week I was searching through a couple hundred gigs small xml files. I found that:
$ LC_ALL=C fgrep -r STRING .
was much faster than plain grep. This was on a CentOS 5 box, so maybe newer versions of grep are smarter.But then again, if I was on a newer box I'd just install and use ack or ag.
Re:Command-line tools can be faster than your Hadoop cluster
by paralelogram on Sunday January 18, 2015 @06:52PM
(#8909510)
LC_ALL=C makes grep faster because text matching is normally locale-sensitive, for example 'S' ('\x53' in Big5) is not a substring of '兄' ('\xA5\x53' in Big5).