It’s Not Wrong that “ ”.length == 7

Re:It’s Not Wrong that “ ”.length == 7 by sirsinsalot on Friday June 02, 2023 @02:42AM (#36161850)

Exactly this. People conflate unicode with encoding quite a bit. I think it was plan9 and early Go that used "runes" as a unit, where one or more runes formed a character and an array of runes could be encoded into bytes using a given encoding.

The in memory size of a rune was just an implementation detail, and while it could be important for the programmer that the size of a rune was 2 bytes, this didn't mean the length of an array of 2 runes was 4.

I always liked the rune unit, and while my memory is hazy I think it was just code points.

I think part of the issue is programmers and apis mixing bit units for in memory representation of a conceptual value mapping (unicode), conceptual characters, stored size when encoded and so on ... without firming up those abstractions with interfaces. It gets lossy.

[ Parent | on HN ]

Re:It’s Not Wrong that “ ”.length == 7 by eesmith on Friday June 02, 2023 @05:39AM (#36163074)

FWIW, Python uses one of several Unicode string implementation representations, depending on the code points involved:

  >>> import sys
  >>> s = 'A' * 1000
  >>> len(s)
  1000
  >>> sys.getsizeof(s)
  1049
  >>> s = '\N{SNOWMAN WITHOUT SNOW}' * 1000
  >>> len(s)
  1000
  >>> sys.getsizeof(s)
  2074
  >>> s = '\N{MUSICAL SYMBOL G CLEF}' * 1000
  >>> len(s)
  1000
  >>> sys.getsizeof(s)
  4076

[ Parent | on HN ]