Building a high performance JSON parser

Re:Building a high performance JSON parser by zlg_codes on Monday November 06, 2023 @01:29AM (#38159502)

What on Earth are you storing in JSON that this sort of performance issue becomes an issue?

How big is 'large' here?

I built a simple CRUD inventory program to keep track of one's gaming backlog and progress, and the dumped JSON of my entire 500+ game statuses is under 60kB and can be imported in under a second on decade-old hardware.

I'm having difficulty picturing a JSON dataset big enough to slow down modern hardware. Maybe Gentoo's portage tree if it were JSON encoded?

[ Parent | on HN ]

Re:Building a high performance JSON parser by EMM_386 on Monday November 06, 2023 @04:14AM (#38160332)

> What on Earth are you storing in JSON that this sort of performance issue becomes an issue?

I've been in the industry for a while. I've probably left more than one client site muttering "I've seen some things ...".

If it can be done, it will be done. And often in a way that shouldn't have even been considered at all.

Many times, "it works" is all that is needed. Not exactly the pinnacle of software design. But hey, it does indeed "work"!

[ Parent | on HN ]

Re:Building a high performance JSON parser by Edwinr95 on Monday November 06, 2023 @04:33AM (#38160445)

I've seen people dump and share entire databases in JSON format at my job....

[ Parent | on HN ]

Re:Building a high performance JSON parser by nly on Monday November 06, 2023 @04:45AM (#38160512)

I've seen tens of millions of market data events from a single day of trading encoded in JSON and used in various post-trade pipelines.

[ Parent | on HN ]

Re:Building a high performance JSON parser by zlg_codes on Monday November 06, 2023 @10:15PM (#38172771)

Ah, that's a dataset with a size certainly intimidating, and in an environment where performance means money. Thanks for pointing that out!

[ Parent | on HN ]

Re:Building a high performance JSON parser by mxmlnkn on Monday November 06, 2023 @06:22AM (#38161141)

Chrome trace format files also use JSON and can also become large and are a pain to work with.

[ Parent | on HN ]

Re:Building a high performance JSON parser by craigching on Wednesday November 08, 2023 @06:56PM (#38198994)

In my case, sentry events that represent crash logs for Adobe Digital Video applications. I’m trying to remember off the top of my head, but I think it was in the gigabytes for a single event.

[ Parent | on HN ]

Re:Building a high performance JSON parser by pbrumm on Thursday November 09, 2023 @08:44AM (#38204871)

Insurance price transparency can have 16gb of compressed JSON that represents a single object.

Here is the anthem page. The toc link is 16gb

https://www.anthem.com/machine-readable-file/search/

They are complying with the mandate. But not optimizing for the parsers

[ Parent | on HN ]