News for hackers. Stuff that matters.
Re:Analyzing multi-gigabyte JSON files locally by coldtea on Saturday March 18, 2023 @07:36PM (#35214333)
>JSON is a text format and interning it into proper data structures is likely going to take _less_ space, not more.

Wanna bet?

  >>> import sys
  >>> import json
  >>> data_as_json_string = '{"a": 5000, "b": 1000}'
  >>> len(data_as_json_string)
  22
  >>> data_as_native_structure = json.loads(data_as_json_string)
  >>> sys.getsizeof(data_as_native_structure)
  232
That's not even the whole story, as what's 232 bytes is not the contents of the dict, but just the Python object with the dict metadata. So the total for the struct inside is much bigger than 232 bytes.

A single int wrapped as a Python object can be quite a lot by itself:

>>> sys.getsizeof(1) 28

A binary int64 would be 8 bytes for comparison.

Re:Analyzing multi-gigabyte JSON files locally by berkle4455 on Saturday March 18, 2023 @09:00PM (#35214861)
Maybe 2023 is the year people stop using python.
Re:Analyzing multi-gigabyte JSON files locally by IshKebab on Sunday March 19, 2023 @07:34AM (#35217982)
We can dream but I seriously doubt it. Not only are there very few nice alternatives (Dart is great but unpopular; Typescript is great but has a terrible standard library, etc.) but very few of them have a decent REPL which I think is probably Python's most underrated feature.

I mean... it's REPL is extremely basic, but anything you can do in Python you can do in the REPL. Very useful especially for beginners.

Re:Analyzing multi-gigabyte JSON files locally by killingtime74 on Saturday March 18, 2023 @09:14PM (#35214951)
Python is my favorite language but you literally chose one of the least efficient. This would probably be different in Java or Rust.
Re:Analyzing multi-gigabyte JSON files locally by bufferoverflow on Sunday March 19, 2023 @12:16AM (#35215977)
I thought Java stores strings as 16-bit character arrays internally. Which means it's very inefficient when it comes to memory consumption.
Re:Analyzing multi-gigabyte JSON files locally by samus on Sunday March 19, 2023 @04:46AM (#35217168)
Since version 9, Java uses ISO-8859 when possible. Most raw JSON should therefore take up very little space.
Re:Analyzing multi-gigabyte JSON files locally by hsn915 on Sunday March 19, 2023 @01:28AM (#35216313)
I'm pretty sure the GP intended the statement to be for statically typed languages with a struct-like type.
Re:Analyzing multi-gigabyte JSON files locally by coldtea on Sunday March 19, 2023 @05:44PM (#35223992)
It doesn't get much better there:

https://news.ycombinator.com/item?id=35216761