News for hackers. Stuff that matters.
Re:Log Everything as JSON. Make Your Life Easier. by a3_nm on Friday April 27, 2012 @02:55AM (#3897929)
What if I need, say, to find the 10 IPs that make the most requests? With the Apache log format, I can write the following in about 15 seconds:

  cut -d ' ' -f1 log | sort | uniq -c | sort -nr | head
Say you need to follow accesses to a particular file? The following quick and dirty one-liner probably works well enough:

  tail -f log | grep --line-buffered file.pdf
How do you do that with json?

Granted, as soon as your logs stop being a sequence of records (lines) with a fixed sequence of neatly delimited records, you will need something more than text. However, I still don't know of tools to work with json from the command line that are as concise, efficient, flexible and robust as the standard unix utilities for text.

Re:Log Everything as JSON. Make Your Life Easier. by keenerd on Friday April 27, 2012 @06:14AM (#3898378)
> How do you do that with json?

Depends on the schema used, but it would probably be something like

    jshon -a -e "ip" -u < log | sort | uniq -c | sort -nr | head
Re:Log Everything as JSON. Make Your Life Easier. by gravitystorm on Friday April 27, 2012 @06:37AM (#3898424)
The problem with your first command is the "cut" - works great for a while. Then you deal with other_vhosts_combined.log, and there's a leading hostname and port, so you want -f2 instead. After that, you start working with the user-agent field, and after remembering that the date and request fields have internal spaces, settle for something like -f13, which works until someone puts a space in their faked-up referer, and you can't use cut any more.

I love plain-text logs, but that apache logformat is a bad example of "neatly delimited records", given the unstable mixture of spaces between and within fields (some of which are quoted, some not, and one even quoted with square brackets!).

Re:Log Everything as JSON. Make Your Life Easier. by deno on Friday April 27, 2012 @04:26AM (#3898141)
Not JSON, but one vision as to how structured data could fit into UNIX ecosystem: http://xml-coreutils.sourceforge.net/

In your current pipeline just replace `cut` with an XPath expression, everything else stays practically the same.

Of course current XML ecosystem has different tools that makes your case trivial, though that’s something new to learn.

Re:Log Everything as JSON. Make Your Life Easier. by tim_sw on Friday April 27, 2012 @03:02AM (#3897945)
recordStream provides some of these features. Take a look here https://github.com/benbernard/RecordStream