Most of the parsing required for the logfile analysis I describe is written in awk, and goes through roughly 1 million lines of logfile in about 75s on a commodity 2Ghz Intel Xeon.
Pre-caching of host lookups is done via xargs and a suitably high '-j' value. Running on a pre-sorted and de-duped list of IPs, this takes another few minutes. Lookups are read into hash tables for faster processing in the main awk parser (avoids system calls, DNS latency, and especially negative result caching failures).
It's fast and simple. Not a 100% solution, but quick to add bits to over time as new needs arise.
It was the proper add-on to the previous comment kids today don't learn awk. Becuase they have Perl, Ruby, and Python which are supersets of the unix command line tool set. Awk was the king in the years before Perl.
I never really learned to use cut, so I sitll use awk for one-liners the deal with columns in delimited-data text-files.
Or they might have indeed reinvented it. Kids these days and their Heroku! Back in my day we'd process each HTTP request by hand. And we'd do it uphill both ways in the snow.