Command-line tools can be faster than a Hadoop cluster (2014)

Re:Command-line tools can be faster than a Hadoop cluster (2... by londons_explore on Thursday January 30, 2020 @01:55AM (#22189459)

If you always want to search by one thing, you can manually index by that thing. In your case, arrange your log files by the first 6 hex characters of the user ID (/var/log/xxx/xxx/date.log), and grep will typically only then have a few megabytes to scan.

If you need real indexes, or just want something industry standard and maintainable rather than 'some guys grep script', then elastic search is probably the way to go.

[ Parent | on HN ]

Re:Command-line tools can be faster than a Hadoop cluster (2... by Johnny555 on Thursday January 30, 2020 @01:58AM (#22189471)

That was just one thing we needed to search for (but by far the most common). The guy that wrote the parallel grep did try creating some indexes of common fields to speed searches, but quickly realized that he was re-implementing the wheel (poorly)

Plus we made good use of Kibana dashboards for the service

[ Parent | on HN ]