Re:Command-line tools can be faster than a Hadoop cluster (2...
by fxtentacle on Thursday January 30, 2020 @01:45AM
(#22189405)
That tool looks great :) but since we're already seeing <1s search times and the tool is only used by internal support employees, I'm mostly going with "never touch a running system" these days.
While for a database like ES you'd put all of the data into one big pile and then filter by keywords, e.g. host=ftp service=ftp query=IP, for logfiles you usually search on a much smaller set. They are rotated by day and logs are broken down by host and service by rsyslog, so instead of filtering the full 150TB - which is what ES has to do - my grep only needs to look at the 1-2 GB of data inside the file where host, service, and date match.
Re:Command-line tools can be faster than a Hadoop cluster (2...
by pepemon on Thursday January 30, 2020 @09:46AM
(#22191589)
Do you understand something about such thing as ES indices?
Re:Command-line tools can be faster than a Hadoop cluster (2...
by reagent_finder on Thursday January 30, 2020 @02:32AM
(#22189613)
RG is awesome for the basic use case. Using it across platforms just makes you tear your hair out, though.
Re:Command-line tools can be faster than a Hadoop cluster (2...
by foota on Thursday January 30, 2020 @03:06AM
(#22189724)
I'm curious, why? Looks pretty easy to get the binaries.
Re:Command-line tools can be faster than a Hadoop cluster (2...
by burntsushi on Thursday January 30, 2020 @07:08AM
(#22190777)
Why? Can you share your frustrations?
[1] https://github.com/BurntSushi/ripgrep