Re:Analyzing multi-gigabyte JSON files locally
by pkaye on Saturday March 18, 2023 @01:10PM
(#35210935)
How about something like a Go interface which is essentially two pointers and the data itself.
[ Parent
| on HN ]
Re:Analyzing multi-gigabyte JSON files locally
by Groxx on Saturday March 18, 2023 @01:16PM
(#35210999)
That's the same thing they're describing, and equivalent to what Go does for map[string]int64 (plus or minus minor details about how strings are optimized).
The map has its two slices (pointer + length + capacity = 8*3 times two slices) and the string needs a separate allocation somewhere because it's essentially a pointer to a slice of bytes. All of which is true for almost all reasonably efficiency-focused languages, Rust and Go included - it's just how you make a compact hashmap.
Re:Analyzing multi-gigabyte JSON files locally
by Yoric on Saturday March 18, 2023 @02:20PM
(#35211692)
Does Go perform interning of strings?
Re:Analyzing multi-gigabyte JSON files locally
by Groxx on Saturday March 18, 2023 @02:41PM
(#35211871)
* * *
Re:Analyzing multi-gigabyte JSON files locally
by xh-dude on Saturday March 18, 2023 @10:03PM
(#35215252)
Use of Go’s runtime reflection to produce a struct definition would result in one instance of the key string, many instances of the numeric value
This approach is available/implemented in the standard library, the whole story with JSON in Go is includes a few different approaches but this is definitely anticipated.