There are some good arguments against this format but your points were invalid. Key data in many different types of data is much larger than the actual values. E.g ("average_cpu_cylces_per_min": 60)

Gzip or some other compression scheme takes care of that pretty well. It's not rare to do JSON over gzipped HTTP requests.

You gzip csv and csj as well so this is orthogonal to the point.

Is it? I think his point is that key repition may not be a relevant factor when a gzip can be presumed.

is there necessarily a need to repeat key values over lines?

why not have all possible object definitions in the header and data as a single huge array?

You're describing the type-length-value format, used in some places in digital video or in Open Sound Control.

Basically you have a TYPE (fixed width, eg. 16 bits), a LENGTH (again fixed) and a VALUE which is simply a stream of bytes that goes on for LENGTH bytes and is interpreted based on what TYPE it is. You can nest your fields and get very creative in quite a compact way.

One could represent a JSON document with TLV by adding an associative map between the field names and their type numbers, then stream away.

I always loved the simplicity of TLV. Though nowadays, I'm not sure this has much of an advantage over gzipped JSON.

Off-topic: Howdy, Mr. Thénot! Crazy reading the comments and seeing your username here. Hope life is goin' well for ya with the HealthNinja stuff -- haven't seen you since I left Wellcentive (around the same time as mbonnette, if that gives you an idea of who I am).

Because to parse it legally you'd have to store the entire file in memory. That's not actually streaming. The CSJ format puts the object schema and order up front and just applies it to each line afterwards. Stream away...