Re:Log Everything as JSON. Make Your Life Easier.
by keenerd on Friday April 27, 2012 @06:14AM
(#3898378)
> How do you do that with json?
Depends on the schema used, but it would probably be something like
jshon -a -e "ip" -u < log | sort | uniq -c | sort -nr | head
Re:Log Everything as JSON. Make Your Life Easier.
by gravitystorm on Friday April 27, 2012 @06:37AM
(#3898424)
The problem with your first command is the "cut" - works great for a while. Then you deal with other_vhosts_combined.log, and there's a leading hostname and port, so you want -f2 instead. After that, you start working with the user-agent field, and after remembering that the date and request fields have internal spaces, settle for something like -f13, which works until someone puts a space in their faked-up referer, and you can't use cut any more.
I love plain-text logs, but that apache logformat is a bad example of "neatly delimited records", given the unstable mixture of spaces between and within fields (some of which are quoted, some not, and one even quoted with square brackets!).
Re:Log Everything as JSON. Make Your Life Easier.
by deno on Friday April 27, 2012 @04:26AM
(#3898141)
Not JSON, but one vision as to how structured data could fit into UNIX ecosystem: http://xml-coreutils.sourceforge.net/
In your current pipeline just replace `cut` with an XPath expression, everything else stays practically the same.
Of course current XML ecosystem has different tools that makes your case trivial, though that’s something new to learn.
Re:Log Everything as JSON. Make Your Life Easier.
by tim_sw on Friday April 27, 2012 @03:02AM
(#3897945)
recordStream provides some of these features. Take a look here https://github.com/benbernard/RecordStream
Granted, as soon as your logs stop being a sequence of records (lines) with a fixed sequence of neatly delimited records, you will need something more than text. However, I still don't know of tools to work with json from the command line that are as concise, efficient, flexible and robust as the standard unix utilities for text.