Twitter (disclosure: I manage/was early engineer on the analytics infra team) used Json initially, and it quickly turned into a mess because JSON does not enforce schemas or type safety. There is no way for your consumers to find out what you are logging, other than to sample records. This makes evolving logging, deprecating fields, and doing sanity checking very complex (you basically have to build a separate metadata discovery system).
Use Thrift or Avro. Avro is extra nice because not only does it have a schema, it keeps the schema together with the data; however, the support for it is not as mature/wide. It's improving fast, though. Thrift, and, I think, Avro, have JSON protocols if you really love JSON -- you get the human-readable debugging mode, the metadata, the type safety, and compact binary representation.. happiness ensues.
Only if the text representation is pretty-printed in a certain way.
> It's hard to do that with XML because even if you find the right user, the entire log message/object spans multiple lines.
XML tooling makes grepping obsolete.
cat log.xml | xpath //log[@user="bob smith"]Check out xmllint too, ESP the "shell" feature in it (basically a REPL for xpath and other useful XML spelunking.)
DOM is completely optional, and you only need it for convenience or to observe the modifications, e.g. the way browsers make use of it.
Just think how you would compile any XPath expression — it’s very similar to REGEX, only for structured data.
Worst case scenario XPath expression will not yield any results before traversing the entire tree. However any XPath 1.0 (and I imagine 2.0 is no different in that regard) can be compiled into a deterministic state machine (DFA), which only needs to keep tabs on how many elements it has seen and what conditions has been met.
What XPath 1.0 specifically doesn’t allow are arbitrary sub-expressions in predicates. Those would be problematic in certain conditions.
The only potential issue with performance is when running a large number of XPEs against a single stream. So there are various techniques to remedy that, including merging state machines for branching expression etc.
The JSON spec is very precise, you can find it right on the front page of http://json.org/.
<foo grep bar | ...
if you really want L-to-R whilst avoiding the redundant cat and its overhead. That doesn't allow grep to do its speed-ups though, because it's still reading stdin, not a filename.Sidenote: benbernard worked on Amazon dev tools team
1) As others said, you should definitely use a library for JSON reading/writing.
2) JSON is simple and surprisingly fast to read/write. In one large benchmark (http://code.google.com/p/thrift-protobuf-compare/wiki/Benchm...), the Jackson JSON library (for Java) comes in somewhere between Thrift and Protocol Buffers.
3) JSON feels friendlier than protocol buffers and Thrift because it's human readable instead of binary.
4) JSON is more convenient than multi-line formats like XML because you can grep for things easily. For example, if you have {time:015128752, event:"authentication error", user:"bob smith", details: "..."}, then I can take do the following:
It's hard to do that with XML because even if you find the right user, the entire log message/object spans multiple lines.5) JSON is not as compact as binary, but it gets surprisingly close if you gzip it. All of those repeated descriptive attribute names are great for humans, but compression algorithms love them as well.
6) The format is fairly universal, but if you ever come across a language without JSON parsing libraries, the format is not so hard that you couldn't write your own parser (compare that to Thrift, XML, etc)