News for hackers. Stuff that matters.
Re:Company withdrawing from Facebook as analytics show 80% o... by dredmorbius on Monday July 30, 2012 @06:22PM (#4314989)
Indeed.

Most of the parsing required for the logfile analysis I describe is written in awk, and goes through roughly 1 million lines of logfile in about 75s on a commodity 2Ghz Intel Xeon.

Pre-caching of host lookups is done via xargs and a suitably high '-j' value. Running on a pre-sorted and de-duped list of IPs, this takes another few minutes. Lookups are read into hash tables for faster processing in the main awk parser (avoids system calls, DNS latency, and especially negative result caching failures).

It's fast and simple. Not a 100% solution, but quick to add bits to over time as new needs arise.

There are no comments on this comment yet.