News for hackers. Stuff that matters.
Re:Command-line tools can be faster than a Hadoop cluster (2... by fxtentacle on Thursday January 30, 2020 @01:45AM (#22189405)
That tool looks great :) but since we're already seeing <1s search times and the tool is only used by internal support employees, I'm mostly going with "never touch a running system" these days.

While for a database like ES you'd put all of the data into one big pile and then filter by keywords, e.g. host=ftp service=ftp query=IP, for logfiles you usually search on a much smaller set. They are rotated by day and logs are broken down by host and service by rsyslog, so instead of filtering the full 150TB - which is what ES has to do - my grep only needs to look at the 1-2 GB of data inside the file where host, service, and date match.

Re:Command-line tools can be faster than a Hadoop cluster (2... by pepemon on Thursday January 30, 2020 @09:46AM (#22191589)
Do you understand something about such thing as ES indices?