Re:How I cut GTA Online loading times by 70%
by Slikey on Sunday February 28, 2021 @06:29PM
(#26298129)
Check out https://github.com/simdjson/simdjson
More than 3 GB/s are possible. Like you said 10 MB of JSON is a breeze.
Re:How I cut GTA Online loading times by 70%
by tyingq on Sunday February 28, 2021 @04:59PM
(#26297496)
The JSON patch took out more of the elapsed time. Granted, it was a terrible parser. But I still think JSON is a poor choice here. 63k x X checks for colons, balanced quotes/braces and so on just isn't needed.
Time with only duplication check patch: 4m 30s
Time with only JSON parser patch: 2m 50s
Re:How I cut GTA Online loading times by 70%
by masklinn on Monday March 01, 2021 @02:16AM
(#26300402)
> But I still think JSON is a poor choice here.
It’s an irrelevant one. The json parser from the python stdlib parses a 10Mb json patterned after the sample in a few dozen ms. And it’s hardly a fast parser.
Down a little in the article and you'll see one of the real issues:
> But before it’s stored? It checks the entire array, one by one, comparing the hash of the item to see if it’s in the list or not. With ~63k entries that’s (n^2+n)/2 = (63000^2+63000)/2 = 1984531500 checks if my math is right. Most of them useless.