News for hackers. Stuff that matters.
Re:Command-line tools can be faster than a Hadoop cluster (2... by bobnamob on Thursday January 30, 2020 @01:38AM (#22189375)
Might be worth taking a look at ripgrep[1].

[1] https://github.com/BurntSushi/ripgrep

Re:Command-line tools can be faster than a Hadoop cluster (2... by fxtentacle on Thursday January 30, 2020 @01:45AM (#22189405)
That tool looks great :) but since we're already seeing <1s search times and the tool is only used by internal support employees, I'm mostly going with "never touch a running system" these days.

While for a database like ES you'd put all of the data into one big pile and then filter by keywords, e.g. host=ftp service=ftp query=IP, for logfiles you usually search on a much smaller set. They are rotated by day and logs are broken down by host and service by rsyslog, so instead of filtering the full 150TB - which is what ES has to do - my grep only needs to look at the 1-2 GB of data inside the file where host, service, and date match.

Re:Command-line tools can be faster than a Hadoop cluster (2... by pepemon on Thursday January 30, 2020 @09:46AM (#22191589)
Do you understand something about such thing as ES indices?
Re:Command-line tools can be faster than a Hadoop cluster (2... by reagent_finder on Thursday January 30, 2020 @02:32AM (#22189613)
RG is awesome for the basic use case. Using it across platforms just makes you tear your hair out, though.
Re:Command-line tools can be faster than a Hadoop cluster (2... by foota on Thursday January 30, 2020 @03:06AM (#22189724)
I'm curious, why? Looks pretty easy to get the binaries.
Re:Command-line tools can be faster than a Hadoop cluster (2... by burntsushi on Thursday January 30, 2020 @07:08AM (#22190777)
Why? Can you share your frustrations?