News for hackers. Stuff that matters.
Re:Log Everything as JSON. Make Your Life Easier. by jacques_chester on Thursday April 26, 2012 @09:02PM (#3896933)
One of the non-functional requirements of logs is that they should be fast to write. Marshalling data into a structured format takes longer than spitting out sprintfs.

If you really need structure for ease of querying, you might as well go all the way and throw it into a proper data store.

Re:Log Everything as JSON. Make Your Life Easier. by gorset on Thursday April 26, 2012 @09:20PM (#3897001)
Remember we're talking about writing to disk. The time it takes to marshal the data is pretty much insignificant to the time it takes to do the actual write.

We use JSON on disk heavily in several places where we generate or receive huge amounts of data. JSON-in-a-file is a pretty good datastore for sequential data processing because it's so convent and you can work with the data using cat, tail or any tool supporting JSON.

Re:Log Everything as JSON. Make Your Life Easier. by jacques_chester on Thursday April 26, 2012 @10:05PM (#3897148)
> The time it takes to marshal the data is pretty much insignificant to the time it takes to do the actual write.

And JSON is more verbose.

Basically this is another read/write dichotomy.

When do we pay the price of structuring data? At write time? Or at read time? I'd rather pay it at read time, as writing may be shaving performance from my actual primary production system.

Re:Log Everything as JSON. Make Your Life Easier. by gorset on Friday April 27, 2012 @04:32AM (#3898150)
This is premature optimization. Do you think it's expensive to write a few brackets, quotes and escape a few characters? We have harder problems to think about than optimizing for writes in our logging system :-)

How can you trade write/read without some structure in the log message? And do you know that sprintf actually costs something too? It's basically a mini-language which must be parsed and interpreted. You can go google sprintf+performance to see stories of people finding this out - but most of the time it doesn't matter, because the cost is insignificant and you can do what's most convenient for you.

Re:Log Everything as JSON. Make Your Life Easier. by jacques_chester on Saturday April 28, 2012 @11:57PM (#3905032)
My point is that structure is imposed somewhere if you want to query your data. Either that structure is in the data, or in tools that parse the data. There is a "conservation of structure", if you like.

Most of the bunfights between proponents of different technologies is really an argument about where to pay structural tax. You can pay more at write time or read time on a sliding scale.

For logs, I think the smart option is to pay as little write-time overhead as possible. Their purpose is to maximally describe an event with minimal interruption to service. Every strictly unnecessary adornment to structure takes you further away from that core non-functional goal.

Re:Log Everything as JSON. Make Your Life Easier. by stevewilhelm on Thursday April 26, 2012 @10:03PM (#3897140)
You're still logging to local disk?
Re:Log Everything as JSON. Make Your Life Easier. by thezilch on Thursday April 26, 2012 @10:35PM (#3897228)
a) log writes should be buffered b) once buffered or immediately (ex. UDP [ex. Etsy's StatsD]), the client can typically continue and even complete without the log being flushed all the way to non-volatile media
Re:Log Everything as JSON. Make Your Life Easier. by frsyuki on Thursday April 26, 2012 @09:17PM (#3896992)
Spitting out sprintfs means parsing texts. JSON might be slow since it's text, but binary based formats should be faster especially the log includes many integers.
Re:Log Everything as JSON. Make Your Life Easier. by jacques_chester on Thursday April 26, 2012 @10:03PM (#3897139)
sprintf tends to be a faster path than "build tree, render down to text".
Re:Log Everything as JSON. Make Your Life Easier. by deno on Friday April 27, 2012 @01:58AM (#3897775)
Both XML and JSON can be streamed. No tree building necessary.