News for hackers. Stuff that matters.
Re:Memory use in CPython and MicroPython by masklinn on Sunday July 02, 2017 @06:12AM (#14681818)
> Hint: this is much closer to what string handling in sensibly designed languages like Rust and Go or Erlang looks like.

* Rust has a very strict split between bytes and text

* Go has fucked up bytestrings which depending on validity may break at any point depending on the API you're applying

* Erlang's strings are widely considered awful, and are literally just linked lists of integers making them slow, memory-heavy and prone to pretty-printer whims (as the pretty printer decides on the fly whether a given list looks like a string — and should be printed thus — or does not — and should be printed as a regular list of integers)

Re:Memory use in CPython and MicroPython by patrec on Sunday July 02, 2017 @01:03PM (#14683283)
Erlang's strings might be widely considered awful, but probably not by people who actually have a clue – the numbers thing is ugly, but overall it's a very nice design and has very nice features that are pretty much unique to Erlang.

Erlang has 3 types of strings:

1. the list of integers thing, which you should mostly avoid

2. binary strings (like python2 str or go)

3. iolists: a list or element of either of 1, 2 or 3. This is the nice part – it basically means erlang has ropes. And all the "egress" functions (sending udp or tcp packets or writing to stdout or files or sockets etc.) take iolists. So you can avoid a lot of pointless copying and write text or binary data very efficiently; I know of no other (non-BEAM) language that has similarly good support for this, do you?

See http://www.evanmiller.org/elixir-ram-and-the-template-of-doo... for more details.

Re:Memory use in CPython and MicroPython by masklinn on Sunday July 02, 2017 @03:21PM (#14683919)
> Erlang's strings might be widely considered awful, but probably not by people who actually have a clue – the numbers thing is ugly, but overall it's a very nice design

It is nice in a theoretical sense, in the same way Haskell's [Char] is nice, in a practical sense it's awful.

> 2. binary strings (like python2 str or go)

AKA not actually strings, I don't remember Erlang having much if any string-manipulation API for working on bytes.

> 3. iolists: a list or element of either of 1, 2 or 3. This is the nice part – it basically means erlang has ropes. And all the "egress" functions (sending udp or tcp packets or writing to stdout or files or sockets etc.) take iolists.

This is only for output, so they're not actually ropes.

> So you can avoid a lot of pointless copying and write text or binary data very efficiently; I know of no other (non-BEAM) language that has similarly good support for this, do you?

Most mutable languages will use buffered output streams and possibly some sort of string sequence (even C has iovecs), so while iolists are a good fit for Erlang they make sense in few other languages.

Re:Memory use in CPython and MicroPython by patrec on Sunday July 02, 2017 @04:22PM (#14684166)
> It is nice in a theoretical sense, in the same way Haskell's [Char] is nice, in a practical sense it's awful.

Haskell's list of Char is awful in both a theoretical and practical sense whereas in erlang's case you can think of it as just a degenerate iolist.

> I don't remember Erlang having much if any string-manipulation API for working on bytes

Elixir is better but erlang certainly has string manipulation functionality for binaries: pattern matching, regular expressions, search, splitting and joining etc.

> This is only for output, so they're not actually ropes.

Not true, lots of non-output functions also accept iolists, e.g. the re module.

Re:Memory use in CPython and MicroPython by patrec on Sunday July 02, 2017 @01:15PM (#14683341)
BTW, I agree that you want to differentiate between bytes and text, but the underlying representation should be utf-8 bytes, not some constant time indexable utf-32, which is a semantics and performance disaster. It's much better to mess up in not having a clear distinction between bytes and text (annoying, but mostly easy to work around) then to mess up by having constant-time indexable utf-32 as python3 does (not at all easy to work around and even the bits that are easy to work around, like pretending stdin or stdout and sys.argv are unicode on unix are still super-annoying).
Re:Memory use in CPython and MicroPython by masklinn on Sunday July 02, 2017 @03:24PM (#14683927)
> BTW, I agree that you want to differentiate between bytes and text, but the underlying representation should be utf-8 bytes

Sure, but for most languages older than a few years this ship has sadly long sailed and few would be willing to switch to O(n) indexing.

> It's much better to mess up in not having a clear distinction between bytes and text (annoying, but mostly easy to work around)

Except when it's not and the packages you're trying to use simply do not work and fuck up mightily with non-ascii data.

> then to mess up by having constant-time indexable utf-32 as python3 does

It's funny that just above you praise Erlang which does exactly that. Meanwhile Python 3.3 introduced a flexible string representation so depending on the content of the string the internal string encoding may be ISO-8859-1, UCS2 or UCS4 (and yes sadly only these as the designers refused to break the constant-time indexing guarantee).

Re:Memory use in CPython and MicroPython by patrec on Sunday July 02, 2017 @04:24PM (#14684172)
Where's erlang's constant time indexable utf-32 string?
Re:Memory use in CPython and MicroPython by masklinn on Sunday July 02, 2017 @06:19PM (#14684640)
Erlang's strings are even worse than that since they're non-constant time indexable UTF-32.
Re:Memory use in CPython and MicroPython by dom0 on Sunday July 02, 2017 @09:56AM (#14682502)
> Erlang

What the fuck. And I thought constant-time indexable strings were a bad idea!

Re:Memory use in CPython and MicroPython by masklinn on Sunday July 02, 2017 @11:03AM (#14682784)
FWIW it's way too common in functional languages, Haskell's standard String is a typedef for [Char] (a linked list of Char) hence the proliferation of various other string types.
Re:Memory use in CPython and MicroPython by patrec on Sunday July 02, 2017 @12:51PM (#14683235)
They are. You want to use iolists in Erlang. Constant time concatenation (which is useful) rather than constant time indexing into codepoints (which is not; and erlang does not have BTW).