Re:Ask HN: what software do you use for analyzing apache logs?
by sam on Tuesday April 08, 2008 @01:25AM
(#157599)
apachelog is a nice python module for parsing log lines from apache. it works as a great base for doing your own analysis.
http://code.google.com/p/apachelog/
edit:
I use cronolog to break up the logfiles daily and then I run a python script (which uses apachelog to handle the nasty parsing) to create a summary dictionary of parameters from that day. For example:
{num_unique_ips:140, num_pageviews:532, ...}
I pickle that dictionary and save it as a file. So every day has a raw log file and a "summary dictionary" file. To make plots I go through the summary files and unpickle them to extract the quantities of interest.
It's based on this perl module: http://cpan.uwinnipeg.ca/~peterhi/Apache-LogRegex
To manage log files, use cronolog
edit: I use cronolog to break up the logfiles daily and then I run a python script (which uses apachelog to handle the nasty parsing) to create a summary dictionary of parameters from that day. For example:
{num_unique_ips:140, num_pageviews:532, ...}
I pickle that dictionary and save it as a file. So every day has a raw log file and a "summary dictionary" file. To make plots I go through the summary files and unpickle them to extract the quantities of interest.