News for hackers. Stuff that matters.

Ask HN: When do you index your logs?

Do you write your log-worthy events to a file, and then parse them when you need to analyze them (e.g., Splunk); or do you write the structured event to a structured data store, such as a SQL database, which maintains indices in real time? Both approaches can be configured to perform the parsing and indexing on a remote node.
( 1 comment | 4 points | on HN )
Re:Ask HN: When do you index your logs? by ScottWhigham on Thursday June 06, 2013 @01:35PM (#5833839)
I do it nightly to a SQL database. It's my morning routine to review the logs, add firewall rules based on various "things" I find, or order new/replace equipment based on what I see. I have a heavily partitioned SQL database and so my load times for one day are very fast (< 3 mins typically).