Re:Robust APIs Are Weird
by Felk on Saturday June 20, 2020 @06:54AM
(#23583196)
JSON generally requiring the entire document to be present before parsing can indeed be a problem. However, for one common use case there is a pseudo-standard, namely "JSON lines".
"JSON lines" extends JSON by
a) not allowing newlines in values and
b) allowing newlines to be used as a top-level separator for elements of a list, instead of the JSON-compliant comma-separated list enclosed in square brackets
This allows for per-line parsing, which can significantly reduce memory usage and allow for partial parsing in some scenarios. I've seen it used where the data is fundamentally a large collection at the top level where producing proper JSON was using too much memory
"JSON lines" extends JSON by a) not allowing newlines in values and b) allowing newlines to be used as a top-level separator for elements of a list, instead of the JSON-compliant comma-separated list enclosed in square brackets
This allows for per-line parsing, which can significantly reduce memory usage and allow for partial parsing in some scenarios. I've seen it used where the data is fundamentally a large collection at the top level where producing proper JSON was using too much memory