News for hackers. Stuff that matters.

A comment on Memory use in CPython and MicroPython. Parent comment

Re:Memory use in CPython and MicroPython by patrec on Sunday July 02, 2017 @01:15PM (#14683341)
BTW, I agree that you want to differentiate between bytes and text, but the underlying representation should be utf-8 bytes, not some constant time indexable utf-32, which is a semantics and performance disaster. It's much better to mess up in not having a clear distinction between bytes and text (annoying, but mostly easy to work around) then to mess up by having constant-time indexable utf-32 as python3 does (not at all easy to work around and even the bits that are easy to work around, like pretending stdin or stdout and sys.argv are unicode on unix are still super-annoying).
[ Parent | on HN ]
Re:Memory use in CPython and MicroPython by masklinn on Sunday July 02, 2017 @03:24PM (#14683927)
> BTW, I agree that you want to differentiate between bytes and text, but the underlying representation should be utf-8 bytes

Sure, but for most languages older than a few years this ship has sadly long sailed and few would be willing to switch to O(n) indexing.

> It's much better to mess up in not having a clear distinction between bytes and text (annoying, but mostly easy to work around)

Except when it's not and the packages you're trying to use simply do not work and fuck up mightily with non-ascii data.

> then to mess up by having constant-time indexable utf-32 as python3 does

It's funny that just above you praise Erlang which does exactly that. Meanwhile Python 3.3 introduced a flexible string representation so depending on the content of the string the internal string encoding may be ISO-8859-1, UCS2 or UCS4 (and yes sadly only these as the designers refused to break the constant-time indexing guarantee).

[ Parent | on HN ]
Re:Memory use in CPython and MicroPython by patrec on Sunday July 02, 2017 @04:24PM (#14684172)
Where's erlang's constant time indexable utf-32 string?
[ Parent | on HN ]
Re:Memory use in CPython and MicroPython by masklinn on Sunday July 02, 2017 @06:19PM (#14684640)
Erlang's strings are even worse than that since they're non-constant time indexable UTF-32.
[ Parent | on HN ]