News for hackers. Stuff that matters.
Re:Documentation for the JSON Lines text file format by larkinrichards on Sunday February 25, 2024 @02:05AM (#39498366)
Re:Documentation for the JSON Lines text file format by Towaway69 on Sunday February 25, 2024 @02:07AM (#39498374)
Yes -

> Two terms for equivalent formats of line-delimited JSON are:

> Newline delimited (NDJSON)[4] - The old name was Line delimited JSON (LDJSON).[5]

> JSON lines (JSONL)[6]

https://en.wikipedia.org/wiki/JSON_streaming

Re:Documentation for the JSON Lines text file format by rickette on Sunday February 25, 2024 @02:10AM (#39498386)
Pretty much, discussion: https://github.com/ndjson/ndjson.github.io/issues/1

Also when you're using this with GeoJSON there's https://stevage.github.io/ndgeojson/ which has an actual RFC (https://datatracker.ietf.org/doc/html/rfc8142)

Re:Documentation for the JSON Lines text file format by zerocrates on Sunday February 25, 2024 @02:22AM (#39498437)
Which itself is based on an earlier RFC for not-specifically-Geo-JSON, RFC 7464. Both do things a little different than the others: they use the "record separator" character at the start of each line and actually split on that separator when parsing.

The GeoJSON one pretty much seems to exist just to hang an "application/geo+json-seq" media type registration off of. Part of me wants to say this really should have been more of a "all json subtypes are also json-seq subtypes" situation but maybe that's not really feasible with the standards/registration processes.

Re:Documentation for the JSON Lines text file format by o11c on Sunday February 25, 2024 @03:04AM (#39498662)
It's strictly worse, actually.

ndjson specifies sane newline handling, since it works with terminators.

jsonlines works with separators and thus fails to detect truncated values. As a result, it can silently produce incorrect numeric values.

Re:Documentation for the JSON Lines text file format by exyi on Sunday February 25, 2024 @03:34AM (#39498834)
This is only an issue with plain numbers, however. If you'd have the number in an object or array, you'd detect the truncation just as well. Since using jsonl for a plain list of numbers is... overkill, I'd say it's not an issue.

On the other hand, requiring line terminators in the standard would inevitably lead to incompatibility issues. Most software would accept unterminated files, because text libraries do; and so some files will not be terminated. Some applications do not line-terminate files even on Linux (hello VSCode), and it would be even more problematic on Windows

Re:Documentation for the JSON Lines text file format by basil-rash on Sunday February 25, 2024 @04:35AM (#39499254)
VSCode has the ‘files.insertFinalNewline‘ setting to configure this.
Re:Documentation for the JSON Lines text file format by wruza on Sunday February 25, 2024 @04:57AM (#39499405)
It makes it incompatible with concatenative streaming, requires O(n^2) reparsing on every new chunk instead of just scanning for \n.

And if you have to parse values to detect end of record anyway, there’s no point in having jsonl standard at all, since you can just try to parse until the matching brace and repeat on success.

Re:Documentation for the JSON Lines text file format by Kwpolska on Sunday February 25, 2024 @04:31AM (#39499226)
What’s the difference between terminators and separators here? The ndjson spec [0] doesn’t say anything like that, and it seems that ndjson and jsonlines are identical in what documents they accept.

[0]: https://github.com/ndjson/ndjson-spec

Re:Documentation for the JSON Lines text file format by nasso_dev on Sunday February 25, 2024 @05:03AM (#39499443)
im guessing: all values must always end with a terminator, but a separator doesn't need to be present after the last value

i.e. a documemt without a newline at the end is valid jsonl, but invalid ndjson

Re:Documentation for the JSON Lines text file format by yencabulator on Monday February 26, 2024 @10:29PM (#39519823)
A separator separates two records, where as a terminator terminates a record.

You can detect and error out if you see an unterminated record at end of transmission. With separators, the producer might not put a separator after the last record, because there's nothing to separate it from there.

There's no justification for not using terminators, it's just a bad spec. Unsurprising story: the variant with better marketing has less technical chops.