Re:Use streaming JSON to reduce latency on mobile
by ivanhoe on Monday February 19, 2018 @05:14PM
(#16416192)
Jsonlines is also an excellent intermediary format for working with any huge datasets that are just too big to be loaded into the memory all at once, and where data is hierarchical so csv is not an option. Compared to XML you save a lot in the total file size, and it's just as easy to parse.
Re:Use streaming JSON to reduce latency on mobile
by beau on Monday February 19, 2018 @12:40PM
(#16414281)
It's also easy to load them into Google's BigQuery: https://cloud.google.com/bigquery/docs/loading-data#supporte...
Re:Use streaming JSON to reduce latency on mobile
by stingraycharles on Monday February 19, 2018 @01:30PM
(#16414636)
I was just about to name that as an example. Newline separated JSON is a so much simpler solution to the problem than streaming JSON parsers. They're easier to understand, and it's trivial to implement.
From a purity point of view, sure, streaming JSON is more "correct", but I wouldn't call it the better solution.
Re:Use streaming JSON to reduce latency on mobile
by bringtheaction on Monday February 19, 2018 @02:16PM
(#16414972)
Re:Use streaming JSON to reduce latency on mobile
by smnscu on Monday February 19, 2018 @03:19PM
(#16415400)
Google's lead me to this one today, lol: http://ndjson.org/
I'd make fun of NIH but I'm so guilty of it too.
Re:Use streaming JSON to reduce latency on mobile
by laumars on Tuesday February 20, 2018 @01:44AM
(#16418292)
I've just googled what jsonlines was and it turns out i've been using it for a while in log files too where I wanted JSON objects but needed it to behave like random access memory (exactly like you described with grep). I hadn't even realised it was a formal thing. Thank you
Re:Use streaming JSON to reduce latency on mobile
by pkulak on Monday February 19, 2018 @02:01PM
(#16414849)
> That's not the usecase talked about in this article, but it is a usecase that's important to me.
Yeah, and I agree with you 100%. But lot's of things that are great for log storage aren't appropriate for an API.
Re:Use streaming JSON to reduce latency on mobile
by hiccuphippo on Monday February 19, 2018 @02:09PM
(#16414916)
Together with jq, jsonlines make for great logging.
Re:Use streaming JSON to reduce latency on mobile
by beached_whale on Monday February 19, 2018 @01:22PM
(#16414585)
it should be trivial to write a tool to stream a json file with proper newlines
Re:Use streaming JSON to reduce latency on mobile
by greglindahl on Monday February 19, 2018 @01:24PM
(#16414591)
Not sure what you're getting at -- my tools that generate json logfiles do trivially add newlines where I want them.
jq's "compact" mode will reformat files that don't have newlines where I want them.
Re:Use streaming JSON to reduce latency on mobile
by beached_whale on Monday February 19, 2018 @04:21PM
(#16415847)
I mean that the tools to add the newlines to a compact json file with minimal whitespace are not going to be the bottle neck. I am fairly sure that grep uses a forward streaming access pattern and not random access so inserting a relatively simple command into the pipeline to insert newlines, like jq, probably isn't going to slow it down.
The tool itself wouldn't be difficult to write, or like you said, just use jq
With line delimited logfile objects, it's easy to grep for a string of interest and then only parse the lines that match -- much more efficient than parsing an entire logfile to pick out 0.01% of the lines.
That's not the usecase talked about in this article, but it is a usecase that's important to me.