News for hackers. Stuff that matters.
Re:Sonic: A fast JSON serializing and deserializing library by unspecified on Tuesday November 23, 2021 @05:17PM (#29324269)

    Small (400B, 11 keys, 3 layers)
    Medium (13KB, 300+ key, 6 layers)
    Large (635KB, 10000+ key, 6 layers)
I would be so happy to work somewhere where a 0.000635 GB json file is "large".
Re:Sonic: A fast JSON serializing and deserializing library by hiyer on Tuesday November 23, 2021 @09:53PM (#29326486)
From the documentation it seems to me that they're mostly concerned about resource utilization when processing APIs, and not dealing with files. 635KB may be representative of a large API payload in their environment.
Re:Sonic: A fast JSON serializing and deserializing library by BeeOnRope on Tuesday November 23, 2021 @05:21PM (#29324298)
Just another case where a library tests and publishes results for all competing libraries slower than it, but none faster. cough simdjson [1] cough

---

[1] https://github.com/simdjson/simdjson

Re:Sonic: A fast JSON serializing and deserializing library by nemothekid on Tuesday November 23, 2021 @05:36PM (#29324433)
simdjson is listed thought.
Re:Sonic: A fast JSON serializing and deserializing library by memco on Tuesday November 23, 2021 @05:42PM (#29324485)
It is mentioned here: https://github.com/bytedance/sonic/blob/a577eafc253adb943924..., but it isn't included in the benchmarks graphs. Seems this repo is specifically focused on Golang and isn't necessarily motivated by being the fastest JSON [de]serializer on the planet.
Re:Sonic: A fast JSON serializing and deserializing library by craigching on Tuesday November 23, 2021 @05:52PM (#29324582)
There is this one though: https://github.com/minio/simdjson-go

EDIT: Apologies, simdjson-go is included in the benchmarks and discussed as well.

Re:Sonic: A fast JSON serializing and deserializing library by BeeOnRope on Tuesday November 23, 2021 @06:06PM (#29324698)
Well it wasn't in the benchmark graphs and it didn't show in Ctrl+F in the README.

Admittedly, I didn't do an exhaustive search.

Re:Sonic: A fast JSON serializing and deserializing library by TkTech on Tuesday November 23, 2021 @07:14PM (#29325381)
A port of simdjson, simdjson-go, is in the benchmark graphs. It seems to be too fast to be visible in most of the graphs.
Re:Sonic: A fast JSON serializing and deserializing library by skavi on Tuesday November 23, 2021 @08:25PM (#29325951)
the graphs show rate not time
Re:Sonic: A fast JSON serializing and deserializing library by dr_zoidberg on Wednesday November 24, 2021 @06:23AM (#29328895)
Right, they show MB/s of parsed JSON, and it definitely surprised me: simdjson-go is among the slowest! It looks like there's something fishy going on there, as it is _orders of magnitude_ slower that vanilla simdjson.
Re:Sonic: A fast JSON serializing and deserializing library by OJFord on Wednesday November 24, 2021 @05:18AM (#29328562)
If someone runs at a.. 'higher' rate (m/s) than someone else, is it not right to say that they are faster?
Re:Sonic: A fast JSON serializing and deserializing library by tzs on Wednesday November 24, 2021 @07:23AM (#29329280)
Yes. But on a bar graph showing rate taller bars mean faster, but on a bar graph showing time to complete a task taller bars mean slower.

The graphs are showing rate and simdjson-go has shorter bars (or nonexistent bars which suggest a quantity too low to show up at the resolution of the graphs). TkTech though the graphs were showing time and so thought they were showing that simdjson-go is very fast, which skavi is pointing out is reversed because they are showing rate.

Re:Sonic: A fast JSON serializing and deserializing library by OJFord on Wednesday November 24, 2021 @07:58AM (#29329538)
Oh right yes sorry I didn't really tie up the discussion to the graphs (or lacked caffeine or something!).
Re:Sonic: A fast JSON serializing and deserializing library by leftnode on Tuesday November 23, 2021 @06:03PM (#29324671)
I regularly deal with JSON documents several MB in size, but do developers frequently deal with JSON documents several GB in size? If so, where do you encounter something like that? Surely "processing" that much data (for whatever definition of process you have) is orders of magnitude slower than parsing it.

I love the idea of a library trying to squeeze every last bit of performance out of the CPU, but I'm genuinely curious at the problems it solves in the real world.

Re:Sonic: A fast JSON serializing and deserializing library by Groxx on Tuesday November 23, 2021 @06:09PM (#29324716)
Depends. I've had multi-gigabyte `[{..}, {..}, ...]` json arrays from database dumps, and doing even basic things with that with jq takes ages unless you use the (highly obtuse IMO) streaming methods. Sometimes you can pre-grep to filter the results to something trivial to process, but sometimes the structure is not unique enough to let you do that, or it depends on multiple field values - filtering that with a json parser makes perfect sense, and then speed can matter.

That said, a 2x+ improvement for a couple megabytes, especially if done many times per second, is still a significant improvement.

Re:Sonic: A fast JSON serializing and deserializing library by hobs on Tuesday November 23, 2021 @11:28PM (#29326925)
Monthly general ledger entries for the largest real estate companies, tried XML and JSON, eventually landed on compressed CSV for best trade off between human readable large files (~1-3GB) and compressibility.
Re:Sonic: A fast JSON serializing and deserializing library by jeffbee on Tuesday November 23, 2021 @06:10PM (#29324727)
GeoJSON data can be just as big as you’d like. I have a big archive of GIS data where the files are enormous.
Re:Sonic: A fast JSON serializing and deserializing library by magicalhippo on Tuesday November 23, 2021 @07:29PM (#29325512)
Not JSON, but we process XMLs on the order of a GB. Largest ones are consolidated invoices (ie a lot of separate invoices in one file). Other large ones contain rules and codelists in multiple languages.
Re:Sonic: A fast JSON serializing and deserializing library by jonstewart on Tuesday November 23, 2021 @07:39PM (#29325591)
All the time. It's not there's a single record that is that size, but all sorts of things log in json, so you wind up with multi-GB jsonl files. As an example: AWS CloudTrail logs.
Re:Sonic: A fast JSON serializing and deserializing library by nerdponx on Tuesday November 23, 2021 @06:10PM (#29324723)
Huge log files are one use case. Large datasets are another, in data analysis, machine learning, and ETL tasks.
Re:Sonic: A fast JSON serializing and deserializing library by mayama on Wednesday November 24, 2021 @04:35AM (#29328335)
Aren't log files processed a line at a time? Last time I had to deal with some structured log, I streamed lines concurrently into json parser and it went pretty fast.
Re:Sonic: A fast JSON serializing and deserializing library by thechao on Tuesday November 23, 2021 @07:02PM (#29325287)
Hardware config file for a small SOC. It faithfully describes every addressable physical byte.
Re:Sonic: A fast JSON serializing and deserializing library by uncomputation on Tuesday November 23, 2021 @05:29PM (#29324368)
Since when is 635 KB “large”? I suppose it depends on your use case, since they consider 400 B a small, they probably use lots of JSON APIs for many small things.
Re:Sonic: A fast JSON serializing and deserializing library by otabdeveloper4 on Wednesday November 24, 2021 @01:35AM (#29327452)
635kb is indeed "large" if it's one message. (You will have millions of messages in one file if you're doing batch processing.)
Re:Sonic: A fast JSON serializing and deserializing library by greatgib on Tuesday November 23, 2021 @07:02PM (#29325284)
How many libs we have seen saying that they are the fastest deserializing library, until you try to open files with corner cases, like ut8 characters...
Re:Sonic: A fast JSON serializing and deserializing library by cinntaile on Tuesday November 23, 2021 @09:18PM (#29326293)
I thought UTF-8 was the standard encoding that the json spec requires?
Re:Sonic: A fast JSON serializing and deserializing library by kevingadd on Tuesday November 23, 2021 @11:30PM (#29326932)
Because ASCII text is valid UTF-8, you will sometimes encounter "UTF-8" code that doesn't actually work for multi-byte characters
Re:Sonic: A fast JSON serializing and deserializing library by ZephyrBlu on Wednesday November 24, 2021 @12:11AM (#29327109)
Why does it feel like there are consistently new JSON parsing libraries popping up? People joke about JS frameworks, but JSON parsers almost feel like they're at the same level.
Re:Sonic: A fast JSON serializing and deserializing library by harrygeez on Wednesday November 24, 2021 @01:45AM (#29327503)
somehow when I saw it's a fast JSON serializing salary library I knew it's for Go. For some reasons there're always new JSON libraries for it
Re:Sonic: A fast JSON serializing and deserializing library by rizzaxc on Wednesday November 24, 2021 @07:48AM (#29329460)
Go is kinda first class for microservice web app (and probably its only strong point)
Re:Sonic: A fast JSON serializing and deserializing library by snailysnooze on Wednesday November 24, 2021 @02:36AM (#29327720)
web development -> REST -> JSON -> JSON parsing libraries

So like js frameworks, json parsers are everywhere

Re:Sonic: A fast JSON serializing and deserializing library by TheMagicHorsey on Tuesday November 23, 2021 @05:27PM (#29324344)
It seems its mostly written in assembly? I'd be worried about portability, maintainability, and security.

I'd be way more comfortable if it was written in C, or better yet Rust.

Re:Sonic: A fast JSON serializing and deserializing library by scottlamb on Tuesday November 23, 2021 @05:40PM (#29324467)
It sounds like it essentially is written in C. INTRODUCTION.md says:

> As for insufficiency in compiling optimization of go language, we decided to use C/Clang to write and compile core computational functions, and developed a set of asm2asm tools to translate the fully optimized x86 assembly into plan9 and finally load it into Golang runtime.

Github says it's 59.6% assembly and 6.5% C. Possibly the assembly is just the checked-in result of compiling+translating the C?

Gross that they have to do this. I know the Go folks really prioritize speed of compilation, but I wish they'd support debug builds with their own backend and release builds with LLVM so you could get this kind of performance when actually writing Go. I see there have been a few attempts at Go + general-purpose backend (gccgo, llgo, gollvm) but none seem to use the official Go frontend written in Go, so I think they're doomed to be second-class at best.

edit: and/or, if the Go folks and GCC and/or LLVM coulds could negotiate a shared ABI (not necessarily switching to the platform's default C ABI but having "cc --abi=special-golang-stack-copying-thing"), folks could just link against something compiled in C/Rust/whatever without requiring this separate compilation+translation step (the high-maintenance, high-performance path) or CGo overhead (the easy but slow-for-frequent-calls path).

Re:Sonic: A fast JSON serializing and deserializing library by latchkey on Tuesday November 23, 2021 @05:43PM (#29324495)
I really don't care what language it is written in. As long as there is full test coverage, I'm happy.

I've been a long time user of jsoniter and it is much faster than the standard lib. It really makes a difference for the work I do. If this is even faster and has tests, even better.

Re:Sonic: A fast JSON serializing and deserializing library by mastax on Tuesday November 23, 2021 @08:33PM (#29326015)
Using Go assembly lets you avoid the Cgo overhead IIRC.
Re:Sonic: A fast JSON serializing and deserializing library by susodapop on Wednesday November 24, 2021 @12:34AM (#29327195)
Released by ByteDance, who operate Tiktok.
Re:Sonic: A fast JSON serializing and deserializing library by vmception on Wednesday November 24, 2021 @03:02AM (#29327833)
Which probably means its solving a real world and modern problem they encounter and carries the same weight as a library released by Google
Re:Sonic: A fast JSON serializing and deserializing library by bugmen0t on Wednesday November 24, 2021 @09:29AM (#29330268)
Cool. What’s the interop story between Go‘s existing implementation for parsing objects with duplicate keys? Keep first, overwrite or abort?
Re:Sonic: A fast JSON serializing and deserializing library by throwaway889900 on Tuesday November 23, 2021 @05:20PM (#29324289)
Seems like it's similar in performance to Jackson?
Re:Sonic: A fast JSON serializing and deserializing library by asadlionpk on Tuesday November 23, 2021 @08:11PM (#29325834)
I had to try working with a 100GB single json file. Couldn’t find anything that could work with that sized file.
Re:Sonic: A fast JSON serializing and deserializing library by iainmerrick on Wednesday November 24, 2021 @05:42AM (#29328706)
Out of curiosity, what does "working with" mean here? What operations did you need to perform on it? Streaming reads, transforms, indexed reads, appends, edits?

I'm thinking that any general-purpose JSON loader is likely to perform badly for a 100GB file, purely because it'll use 2x (or more likely 10x) as much memory for the parsed representation. So you'd want some kind of special-case reader for huge files -- maybe it just builds some kind of sparse index with pointers back into an mmap of the raw data.

Re:Sonic: A fast JSON serializing and deserializing library by onion2k on Wednesday November 24, 2021 @04:26AM (#29328295)
At that point you might as well ignore the fact it's JSON and just treat it as a stream.
Re:Sonic: A fast JSON serializing and deserializing library by todotask on Wednesday November 24, 2021 @11:09AM (#29331410)
Does it escape data by default? It appear to be a low-priority for the author in the issues.