Re:Parsing gigabytes of JSON per second
by pablodavila on Saturday October 23, 2021 @09:58AM
(#28968561)
Something like UBF might be interesting to you: https://ubf.github.io/ubf/
Re:Parsing gigabytes of JSON per second
by legulere on Saturday October 23, 2021 @09:21AM
(#28968132)
You mean like CBOR?
Re:Parsing gigabytes of JSON per second
by strictfp on Saturday October 23, 2021 @09:39AM
(#28968305)
There's also Minecraft NBT. It's a bit obscure but quite nice. https://minecraft.fandom.com/wiki/NBT_format
Re:Parsing gigabytes of JSON per second
by rstuart4133 on Saturday October 23, 2021 @09:01PM
(#28973912)
Or indeed ASN.1 [0], which started life in 1984 and is still used in a lot of places today.
The idea has been around for almost 40 years in arguably more complete format than any of the above up to and including a standardised schema representation.
It's not the lack of a suitable standard that's holding it back.
Re:Parsing gigabytes of JSON per second
by sitkack on Saturday October 23, 2021 @04:18PM
(#28971858)
You should put your NBTs inside of a netstring, https://en.wikipedia.org/wiki/Netstring
Re:Parsing gigabytes of JSON per second
by SigmundA on Saturday October 23, 2021 @09:33AM
(#28968238)
Sure, looks good, plenty to choose from, just need to get everyone to agree on one.
Although I do like ones that have some concept of a schema to reduce repeated key size and allow validation.
Although it's not a format, the closest thing I have found is Postgres. You get well defined types that make life easier. They are stored efficiently in binary. And have a sensible text representation that means you can still read and write them by hand. I can store a PostGIS geometry with millions of verticies and still use text to create a point or read an envelope. Text formatting is hugely useful and benefits from standardisation. But that does not need to be built into the format.
Re:Parsing gigabytes of JSON per second
by sdfdf4434r34r on Sunday October 24, 2021 @11:28AM
(#28978797)
>{ contentType: "image/jpeg", content: <real binary data not base64> }
It would have to be encoded somehow, because what if the "real binary data" included the byte 0x7D which is '}'
Sorry thats not meant to be a literal format but conceptually mapped to text JSON. It would instead be more like CBOR or other JSON like binary formats at the byte level.
I don't know why the ASCII characters for record and field separators don't get more love. That is what they are there for. My suspicion is because they aren't type-able, so people rarely encounter them.
Re:Parsing gigabytes of JSON per second
by SigmundA on Saturday October 23, 2021 @09:38AM
(#28968299)
Still have the issue of binary data encoding as well since the data will have those bytes in it so they need to be escaped vs a format that defines how to encoding binary as is efficiently (length delimited).
I wish we could all settle on a binary structured format thats not limited to text encoding and more like JSON (key value) that basically everyone uses with real data types (efficiently encoded numbers, dates, binary , etc).
Text files would just be something like { contentType: "text/plain" content : "text here" } while an image could be { contentType: "image/jpeg", content: <real binary data not base64> } you could also add whatever other metadata you want and all of it is retained and easily parseable. Other more structured formats obviously wouldn't just be a blob for content and every system out there would have a structured binary format viewer/editor just like there are text viewers and editors now.
Sqlite is kinda used this way and has some nice properties like indexes and transactions but is relational instead of hierarchal which can be good and bad, not as straight forward to just view or navigate. Protobuf is another used quite a bit now, honestly I don't care just something everyone agree upon that can encode more structure efficiently but still can be easily inspected everywhere.
Doubt this will any time soon but it does seem inevitable in the long run that we figure out a way to send data between system and what a string, number, date etc is and stop having text encoding and escaping issues.