News for hackers. Stuff that matters.
Re:Log Everything as JSON. Make Your Life Easier. by rachelbythebay on Thursday April 26, 2012 @08:56PM (#3896915)
This article feels like it would work just as well with "Protocol Buffers", "Thrift", "XML", or even maybe "ASN.1". If that's truly the case, maybe the better thing to say is "please don't (only) log in ASCII", followed by "please use a format which is hard to get wrong".

JSON scares me a little. Don't you have to worry about escaping a whole bunch of characters just in case something gets the wrong idea about what you have in a field? I saw a page not too long ago which listed about a dozen characters which should be substituted in some manner when used in JSON.

Full disclosure: I got tired of ASCII logging from my web server and wrote something to stream binary protocol buffers (!) to a file instead. http://rachelbythebay.com/w/2012/02/12/progress/

Re:Log Everything as JSON. Make Your Life Easier. by lpolovets on Thursday April 26, 2012 @09:15PM (#3896983)
Over the last 10 years, I've gradually moved from ASCII to Thrift/ProtocolBuffers to JSON. Here are some random thoughts:

1) As others said, you should definitely use a library for JSON reading/writing.

2) JSON is simple and surprisingly fast to read/write. In one large benchmark (http://code.google.com/p/thrift-protobuf-compare/wiki/Benchm...), the Jackson JSON library (for Java) comes in somewhere between Thrift and Protocol Buffers.

3) JSON feels friendlier than protocol buffers and Thrift because it's human readable instead of binary.

4) JSON is more convenient than multi-line formats like XML because you can grep for things easily. For example, if you have {time:015128752, event:"authentication error", user:"bob smith", details: "..."}, then I can take do the following:

  cat log.json | grep 'user:"bob smith"'
It's hard to do that with XML because even if you find the right user, the entire log message/object spans multiple lines.

5) JSON is not as compact as binary, but it gets surprisingly close if you gzip it. All of those repeated descriptive attribute names are great for humans, but compression algorithms love them as well.

6) The format is fairly universal, but if you ever come across a language without JSON parsing libraries, the format is not so hard that you couldn't write your own parser (compare that to Thrift, XML, etc)

Re:Log Everything as JSON. Make Your Life Easier. by squarecog on Thursday April 26, 2012 @10:57PM (#3897292)
JSON is a good choice if you have very tight coupling between producers and consumers of your logs.

Twitter (disclosure: I manage/was early engineer on the analytics infra team) used Json initially, and it quickly turned into a mess because JSON does not enforce schemas or type safety. There is no way for your consumers to find out what you are logging, other than to sample records. This makes evolving logging, deprecating fields, and doing sanity checking very complex (you basically have to build a separate metadata discovery system).

Use Thrift or Avro. Avro is extra nice because not only does it have a schema, it keeps the schema together with the data; however, the support for it is not as mature/wide. It's improving fast, though. Thrift, and, I think, Avro, have JSON protocols if you really love JSON -- you get the human-readable debugging mode, the metadata, the type safety, and compact binary representation.. happiness ensues.

Re:Log Everything as JSON. Make Your Life Easier. by calibraxis on Friday April 27, 2012 @05:55AM (#3898333)
Excellent point. People should read your coworker Nathan Marz's book _Big Data_, for a lucid discussion of schemas.
Re:Log Everything as JSON. Make Your Life Easier. by deno on Thursday April 26, 2012 @10:25PM (#3897198)
> JSON is more convenient than multi-line formats like XML because you can grep for things easily.

Only if the text representation is pretty-printed in a certain way.

> It's hard to do that with XML because even if you find the right user, the entire log message/object spans multiple lines.

XML tooling makes grepping obsolete.

    cat log.xml | xpath //log[@user="bob smith"]
Re:Log Everything as JSON. Make Your Life Easier. by seanp2k2 on Friday April 27, 2012 @12:53AM (#3897602)
+1 for XML. It gets lots of hate, but if you're doing integrations or API work, it's almost everywhere.

Check out xmllint too, ESP the "shell" feature in it (basically a REPL for xpath and other useful XML spelunking.)

Re:Log Everything as JSON. Make Your Life Easier. by matan_a on Friday April 27, 2012 @01:16AM (#3897674)
Just please consider that XPath needs to store the items in a tree structure first (XML needs to be parsed and loaded into memry) before it can be useful. Running that on a large log file would be an interesting performance experiment.
Re:Log Everything as JSON. Make Your Life Easier. by deno on Friday April 27, 2012 @01:30AM (#3897715)
Nonsense, XPath works very well with streamed content, and there are even implementations of XPath engines for FPGA and GPGPU, all of which have very strict memory limitations.

DOM is completely optional, and you only need it for convenience or to observe the modifications, e.g. the way browsers make use of it.

Just think how you would compile any XPath expression — it’s very similar to REGEX, only for structured data.

Re:Log Everything as JSON. Make Your Life Easier. by empthought on Friday April 27, 2012 @09:44PM (#3901762)
I think you're overstating the capabilities and implementation of common XPath processors here, and silently implying the use of a subset or modified version of XPath as specified. Arbitrary XPath will likely require loading the entire document in the worst case, because it allows navigation to any portion of the tree.
Re:Log Everything as JSON. Make Your Life Easier. by deno on Saturday April 28, 2012 @12:50AM (#3902065)
Navigation to any portion of the tree does not imply any need to keep the entire document in memory at once.

Worst case scenario XPath expression will not yield any results before traversing the entire tree. However any XPath 1.0 (and I imagine 2.0 is no different in that regard) can be compiled into a deterministic state machine (DFA), which only needs to keep tabs on how many elements it has seen and what conditions has been met.

What XPath 1.0 specifically doesn’t allow are arbitrary sub-expressions in predicates. Those would be problematic in certain conditions.

The only potential issue with performance is when running a large number of XPEs against a single stream. So there are various techniques to remedy that, including merging state machines for branching expression etc.

Re:Log Everything as JSON. Make Your Life Easier. by empthought on Saturday April 28, 2012 @08:52AM (#3902827)
There's a following:: axis in XPath which lets you look ahead to arbitrary elements. So the processor needs to load the document into memory, which of course is not streaming.
Re:Log Everything as JSON. Make Your Life Easier. by deno on Saturday April 28, 2012 @10:27AM (#3903021)
I don’t see how this axis itself is a problem. Could you provide a specific example?
Re:Log Everything as JSON. Make Your Life Easier. by empthought on Saturday April 28, 2012 @07:06PM (#3904428)
Just /node/node2/following::* in a predicate or whatever. Perhaps you're conflating XPath-as-used with XPath-as-specified? http://msdn.microsoft.com/en-us/library/ms950778.aspx explains that you can't support all of XPath (even 1.0) and process things the way that you describe.
Re:Log Everything as JSON. Make Your Life Easier. by deno on Saturday April 28, 2012 @11:39PM (#3905006)
Right, they completely eliminate any buffering. That’s very strict, but you’re right. I thought you meant that there’s a subset of XPath that can’t work on anything but DOM.
Re:Log Everything as JSON. Make Your Life Easier. by empthought on Sunday April 29, 2012 @10:35AM (#3906049)
Yeah, that's all. The reason it's relevant is that XPath processors often value conformance over expanding the utility of their system. I didn't want someone going off and using libxslt or xalan expecting it to process things in a memory-efficient manner.
Re:Log Everything as JSON. Make Your Life Easier. by LeafStorm on Thursday April 26, 2012 @09:48PM (#3897093)
Fun fact: `user:"bob smith"` is invalid JSON because object keys must always be quoted.

The JSON spec is very precise, you can find it right on the front page of http://json.org/.

Re:Log Everything as JSON. Make Your Life Easier. by sant0sk1 on Thursday April 26, 2012 @10:28PM (#3897208)
OT and pedantic, but common enough that I thought I'd point it out: there's no need to cat and pipe a file into grep, because grep takes a file name as an argument, eg - grep 'user:"bob smith"' log.json
Re:Log Everything as JSON. Make Your Life Easier. by chrisbroadfoot on Thursday April 26, 2012 @11:52PM (#3897439)
I often use cat and pipe to grep, it's nicer to have a left -> right flow.
Re:Log Everything as JSON. Make Your Life Easier. by ralph on Friday April 27, 2012 @05:12AM (#3898240)
I don't like it, but you can do

    <foo grep bar | ...
if you really want L-to-R whilst avoiding the redundant cat and its overhead. That doesn't allow grep to do its speed-ups though, because it's still reading stdin, not a filename.
Re:Log Everything as JSON. Make Your Life Easier. by lindvall on Friday April 27, 2012 @12:10AM (#3897480)
Ends up that passing the file directly to grep will allow it to use tricks to make processing faster than reading from a pipe.
Re:Log Everything as JSON. Make Your Life Easier. by tim_sw on Thursday April 26, 2012 @09:56PM (#3897117)
You can also use tools like recordStream, which allow you to do grep, etc. on individual json fields. https://github.com/benbernard/RecordStream (It has a lot of other features as well)

Sidenote: benbernard worked on Amazon dev tools team

Re:Log Everything as JSON. Make Your Life Easier. by CurtHagenlocher on Friday April 27, 2012 @11:32AM (#3899503)
JSON is no more inherently single-line than XML; both can be formatted to either single-line or multi-line and still be valid.
Re:Log Everything as JSON. Make Your Life Easier. by frsyuki on Thursday April 26, 2012 @09:09PM (#3896963)
There are binary-based serialization formats such as MessagePack [1] or BSON [2] that doesn't need escapeing overhead.

You can simply replace JSON with MessagePack since MessagePack is compatible with JSON.

[1] Why Not MessagePack? http://blog.andrewvc.com/why-arent-you-using-messagepack

[2] Performant Entity Serialization: BSON vs MessagePack (vs JSON) http://stackoverflow.com/questions/6355497/performant-entity...

Re:Log Everything as JSON. Make Your Life Easier. by simonw on Friday April 27, 2012 @02:29AM (#3897846)
Because using a binary format prevents you from tailing your log file without additional tools.
Re:Log Everything as JSON. Make Your Life Easier. by jerf on Thursday April 26, 2012 @10:12PM (#3897162)
"Don't you have to worry about escaping a whole bunch of characters just in case something gets the wrong idea about what you have in a field?"

Either you're using a length-delimited format, which is pretty hostile for humans to read, or you're using a string that will require something to be escaped.

The only third option is bugs, like a newline that gets out into your logs and oops, new log line.

"I saw a page not too long ago which listed about a dozen characters which should be substituted in some manner when used in JSON."

It is very likely what you saw was the list of Unicode characters that are accepted by the JSON format but rejected by browsers and/or HTML in the context of a Javascript embedded in an HTML page. In this case, it's the browsers and/or JS-in-HTML with the issue, not JSON. And I don't mean that as a who-to-blame sort of issue, I mean, if you're producing JSON and consuming JSON and it doesn't pass through HTML it's a non-issue, and any string that passes through JS-in-HTML will have the same problem even if it isn't "JSON", so it really hasn't got anything to do with JSON if I am correct about what you're referring to.

Re:Log Everything as JSON. Make Your Life Easier. by rogerbinns on Thursday April 26, 2012 @11:34PM (#3897382)
I'm happy with JSON and it is the native format of MongoDB, CouchDB and similar databases. And grep/sed/awk. The biggest problem with JSON is that it has no standard way of representing dates, times or binary data.
Re:Log Everything as JSON. Make Your Life Easier. by kelnos on Thursday April 26, 2012 @09:01PM (#3896930)
Don't you have to worry about escaping a whole bunch of characters just in case something gets the wrong idea about what you have in a field?

Yes, but you don't do it yourself. As the article mentions, pretty much every language you'd want to use has a JSON library. Use that, and (bugs aside) it'll take care of escaping for you.

Re:Log Everything as JSON. Make Your Life Easier. by rachelbythebay on Thursday April 26, 2012 @09:04PM (#3896947)
I sure hope everyone uses a library. What worries me is that it looks so simple, so someone might just glue together their own encoding and call it "done".

A higher barrier to entry isn't always a bad thing.

Re:Log Everything as JSON. Make Your Life Easier. by kiyoto on Thursday April 26, 2012 @09:02PM (#3896936)
rachelbythebay: Yea, you've got some good points. I don't think the author means JSON is the be-all-end-all solution to structured logging. It has its issues (escaping for one), but I think it is a reasonable choice because of its wide adoption.

A nice use of protocol buffer there. Speaking of serialization, my personal favorite is MessagePack (httpx://github.com/msgpack).

Re:Log Everything as JSON. Make Your Life Easier. by djcjr on Thursday April 26, 2012 @09:00PM (#3896925)
what, no mention of sexp's?
Re:Log Everything as JSON. Make Your Life Easier. by kiyoto on Thursday April 26, 2012 @09:03PM (#3896942)
djcjr: Are you an OCaml user ;p?