News for hackers. Stuff that matters.
Re:Use streaming JSON to reduce latency on mobile by mikeash on Monday February 19, 2018 @11:52AM (#16413953)
Yeah, that’s really weird. They mention streaming parsers, then immediately say “ignore them.” Why? They apparently think this solution is better somehow, but it would be nice if they’d explain why.

Also, if you’re going to reinvent the wheel and make a custom framing format, why would you choose a delimeter that can legally appear in your content? Separating your JSON with newlines is complete madness. If you’re sending UTF-8 then you can trivially use a byte that cannot appear in the data, like 0xff, as the divider.

Re:Use streaming JSON to reduce latency on mobile by zamalek on Monday February 19, 2018 @03:58PM (#16415686)
> it would be nice if they’d explain why.

Just to fill in the picture here: it's because the built-in parser has a very good chance of being faster than anything you could write[1]. In addition, it's code that you don't own and don't have to maintain.

[1]: with exception to https://news.ycombinator.com/item?id=16413917

Re:Use streaming JSON to reduce latency on mobile by mikeash on Monday February 19, 2018 @04:23PM (#16415856)
You’d use an existing streaming parser, not write your own.
Re:Use streaming JSON to reduce latency on mobile by aaronblohowiak on Monday February 19, 2018 @05:38PM (#16416324)
support for un-marshaling json into objects in a streaming way is spotty / inconvenient. it should be easier than it is.
Re:Use streaming JSON to reduce latency on mobile by icebraining on Monday February 19, 2018 @12:06PM (#16414072)
Newlines can't appear in JSON strings (they must be encoded as \n).
Re:Use streaming JSON to reduce latency on mobile by mikeash on Monday February 19, 2018 @12:08PM (#16414082)
But newlines can appear in JSON anywhere whitespace is allowed otherwise.

Yes, you can say “we will avoid this by never generating JSON with newlines.” But why would you do that when you can easily pick a delimeter that’s guaranteed by the spec not to appear?

Re:Use streaming JSON to reduce latency on mobile by jcrites on Monday February 19, 2018 @03:33PM (#16415509)
I’ve often used the jsonlines format for application logging, though I didn’t have a name for it until today.

I appreciate this format because, as a log file, I can use all of my standard tools to work with it (grep awk etc.). Since it’s a text file, I can use standard text editors to view and search the contents.

Since my application is the one that’s creating the log file, newlines will not appear or will be escaped.

I can often accomplish my goal without parsing the objects, but when I need to, I can break out powerful tools like `jq` or RecordStream to query the contents in a structured way. I can also treat the log file as a streaming or archived data source and process it using database technologies like Redshift or Elasticsearch.

I can’t think of another format that would be more convenient. I have worked with log file formats that are actual XML or JSON documents, and they are inconvenient in practice. I can’t think of another format that would be more convenient than this one for structured application logging.

Re:Use streaming JSON to reduce latency on mobile by IanCal on Monday February 19, 2018 @04:19PM (#16415840)
Lots of other tools work on a line by line basis.
Re:Use streaming JSON to reduce latency on mobile by rhacker on Monday February 19, 2018 @12:32PM (#16414231)
But newlines can't appear in JSON-Lines which is what this is really based off of.
Re:Use streaming JSON to reduce latency on mobile by beau on Monday February 19, 2018 @12:32PM (#16414233)
I made this choice because so many tools support newline-delimited text. A quick search showed that others (Twitter, eBay) made the same choice, so I went with the flow.

*Thank you for NSBlog, it's great!

Re:Use streaming JSON to reduce latency on mobile by mikeash on Monday February 19, 2018 @12:34PM (#16414244)
You’re welcome!

I think the tooling issue is for debugging and inspection, in which case you can just add a quick `tr` invocation to your pipeline to get it into a form they’ll understand.