News for hackers. Stuff that matters.
Re:Why Python 3 Exists by dietrichepp on Thursday December 17, 2015 @12:47PM (#10752971)
I'm getting tired of this argument, we had it in the Rust community as well and I was tired of it there, too. I've seen the same argument over and over again, and every argument had the same hole that I see in your post: What, exactly, is a "character"? Please answer me that, and then you can write a proposal for how the API should work, exactly, and why this works with Indic and German and Korean and everything else.

Strings have to be arrays. It is inevitable. We are on Von Neumann machines with L1 cache lines that are something like 64 bytes wide, so making strings into something other than arrays is a complete non-starter. Because strings are arrays, we use array indexes to slice strings. The indexes are integers, because that just makes sense on a Von Neumann machine.

So your complaint is that you can slice a string in "bad" ways. What makes it bad? You're trying to prevent people from slicing up grapheme clusters, which is a noble goal. But in practice, we get the indexes from methods like str.find() or re.match(), and we treat them as more or less opaque, so we don't end up slicing grapheme clusters very often in practice. Grapheme cluster slicing requires a big Unicode table in memory anyway, so in the rare case that you need it, you can pay the high performance cost for using it. In the meantime, formats like JSON and XML are defined in terms of code points, so using code point arrays eliminates a class of bugs where you could accidentally make malformed XML or JSON, which would then get completely REJECTED by the receiving side, causing your Jabber client to quit or your web browser to show a bunch of mojibake.

And let me ask you this: what do I get when I write:

    x = "a"
    y = "\u0301"
Is the resulting len(x + y) == 1? But len(x) == 1 and len(y) == 1? Can you tell me what the correct behavior is? Might you end up with bugs in programs because len(x + y) != len(x) + len(y)? Or do you introduce extra code points into the string when you concatenate them?

Please, tell me what you actually think is correct behavior. It is far, far more useful than pointing out that something is "wrong" on purely semantic arguments.

Re:Why Python 3 Exists by lucozade on Thursday December 17, 2015 @02:06PM (#10753631)
> Because strings are arrays, we use array indexes to slice strings

But that's a choice and not the only one. Sure you're likely to implement your Unicode string as an array of some type. But that doesn't mean that the only sensible approach is to expose that array directly to the user. Or more specifically as the primary interface.

For my money, Swift has probably the most comprehensive Unicode string API I've seen [0]. What they do is, essentially, have an opaque String type but support various "views" as properties. The main view is called .characters and represents a collection of grapheme clusters. They also have properties that present Unicode scalars, utf8 encoding etc.

Their API is complex, no question. But then proper handling of Unicode is complex. But it does show that there are other options than simply exposing the Unicode scalars.

BTW to your point on len(x) + len(y). The answer is 2 if you define len() on Unicode scalars but 1 if you define it on characters. Why? because len("\u301") should be 0. It's not a character, it's a base modifier. It is, of course, true that getting this right is likely to be substantially more expensive than getting it wrong but that doesn't mean it can't be done.

[0] https://developer.apple.com/library/ios/documentation/Swift/...

Re:Why Python 3 Exists by snuxoll on Thursday December 17, 2015 @06:45PM (#10755568)
Check out Perl 6's strings, they are implemented similarly to the Swift usage of string "views" but with a default that assumes you want NFC with the ability to chose alternate normalizations and raw access to codepoints if you want that instead of grapheme clusters.
Re:Why Python 3 Exists by raiph on Friday December 18, 2015 @12:05PM (#10759468)
> Swift has probably the most comprehensive Unicode string API I've seen

Perl 6 has long specified an interesting and rich approach to Unicode. Have you explored it?

(An initial implementation for an initial release of the Rakudo compiler was worked on over the last few years and declared done, modulo bugs, a few weeks ago. A recent blog post provides a friendly overview.[1].)

One notable difference is that Swift adopts an iteration-only view of strings for dealing with characters whereas Perl 6 does not. One consequence is that indexing operations in Perl 6 are O(1) (time) at the expense of potentially (and for the current implementation, typically) greater use of RAM.

> BTW to your point on len(x) + len(y). The answer is 2 if you define len() on Unicode scalars but 1 if you define it on characters. Why? because len("\u301") should be 0.

Are you 100% sure of this of your last sentence ("should be 0")? I get that it's intuitively reasonable that an isolated base modifier be length 0 but I thought the Unicode standard specified (or at least recommended) that it be 1, and I note that Perl 6 treats isolated base modifiers (or at least this particular one) as length 1:

> ("a").chars 1

> (0x301.chr).chars 1

> ("a" ~ 0x301.chr).chars 1

So both Swift and Perl 6 correctly report the length of "a" as 1 and an "a" concatenated with "\u301" as 1 but disagree on what to do with an isolated base modifier.

> It is, of course, true that getting this right is likely to be substantially more expensive than getting it wrong but that doesn't mean it can't be done.

Indeed.

[1] https://perl6advent.wordpress.com/2015/12/07/day-7-unicode-p...

Re:Why Python 3 Exists by klodolph on Thursday December 17, 2015 @02:45PM (#10753935)
len("\u0301") == 0 is extremely surprising.
Re:Why Python 3 Exists by tomp on Thursday December 17, 2015 @06:48PM (#10755591)
Not really. "\u0301".characters should be equivalent to "".characters, so len("\u0301".characters) == 0 == len("".characters) makes sense.
Re:Why Python 3 Exists by lucozade on Thursday December 17, 2015 @03:53PM (#10754464)
Of course. What you were expecting was 6 right?

You learnt ASCII. Then you learnt about escaping to get around limitations in ASCII. Then you learnt that Unicode got around non-Latin issues with ASCII.

Each step has added cognitive load. It seems surprising to me that the next step wouldn't.

Re:Why Python 3 Exists by klodolph on Thursday December 17, 2015 @04:13PM (#10754587)
The surprise comes with a cost, and the cost is in bugs. The point of Python 3 is to eliminate some bugs in Python 2, and forcing developers to redesign their code around a completely new string API--no matter if that new API is more semantically sound--will come with its own cost of introduced bugs and more painful conversions, which goes against the purpose of Python 3.

I'm not saying that the string API in Python is perfection, but it had a specific role to fill in the larger Python ecosystem, and it does that very well.

Anyway... I would say that len(x) == 0 should be the same as saying that x is empty. According to the Unicode standard, the number of grapheme clusters in "\u0301" is 1, anyway! And on most systems, the string will display as an acute accent in its own space, and you can place the cursor before or after it.

Not like the set of valid "cursor positions" in a string isn't a locale-dependent value, anyway. Grapheme clusters can be thought of to be locale-independent but they are NOT the same as cursor positions.

Re:Why Python 3 Exists by lucozade on Thursday December 17, 2015 @04:53PM (#10754843)
Not to sound too factious but the decision to redesign the string API was a big reason why Python 3 has been a little on the slow cooked side. In for a penny and all that.

If I recall correctly the modifier on it's own isn't correct. There is a base that should go before it to produce a valid character. I don't recall the base unfortunately and I don't have the spec to hand.

Re:Why Python 3 Exists by klodolph on Friday December 18, 2015 @12:00PM (#10759444)
What do you mean by "correct"? The Unicode standard talks about well-formed UTF-8/16/32, but in general, any sequence of code points is permissible. By derivation, any sequence of code points can appear in JSON strings, and any sequence of code points excluding control characters can appear in XML. I personally have read the Unicode spec, and used it to implement some of the algorithms presented. This includes the grapheme cluster breaking algorithm... the case of combining characters at the beginning of a line is, in fact, covered by the spec, and cause a break both before and afterwards. In general, there are other cases where grapheme cluster breaks are not preserved by string concatenation, and you often want a locale-specific version anyway.
Re:Why Python 3 Exists by lucozade on Friday December 18, 2015 @01:13PM (#10759891)
> What do you mean by "correct"?

I mean this from the standard: "Nonspacing combining marks used by the Unicode Standard may be exhibited in apparent isolation by applying them to U+00A0 no-break space. "

You can have valid Unicode that doesn't do this. But it's not intended to represent the character as some kind of textual thing. It's intended so that you can build up Unicode strings sensibly. Same idea as allowing mis-matched surrogates.

Re:Why Python 3 Exists by anon1385 on Thursday December 17, 2015 @01:02PM (#10753088)
The correct answer is that strings shouldn't have length methods. There is no such thing as the 'length' of a string because as you pointed out there is no clear definition of what a 'character' is.

https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-s...

Re:Why Python 3 Exists by ubernostrum on Thursday December 17, 2015 @05:20PM (#10755011)
The correct answer is that strings shouldn't have length methods

There are plenty of uses for getting the length of a string where you will not be running into combining characters, clusters or any other things which would require special handling.

(remember that on the internet, all the data you receive is initially in string form, and very often we want to receive data representing fixed-length values such as phone numbers or credit card numbers, so "you're wrong for wanting to know the length" is both a non-starter and, well, just plain wrong)

Re:Why Python 3 Exists by anon1385 on Friday December 18, 2015 @05:13AM (#10757678)
>There are plenty of uses for getting the length of a string where you will not be running into combining characters, clusters or any other things which would require special handling.

How do you know that? Any user supplied text may contain those things.

>remember that on the internet, all the data you receive is initially in string form

It's not. It's bytes.

>very often we want to receive data representing fixed-length values such as phone numbers or credit card numbers

This is a good example of why strings shouldn't have a length property - people don't understand what length actually does in most languages. If you think that a len() function can verify that the contents of a text field is in the format of a CC number then you don't understand the len function. It can't do what you are asking. See the examples given in the GP (len("\u0301") for example).

The fact that the people who advocate for length functions want to use them for things that they are broken for is exactly why length functions shouldn't exist.

Re:Why Python 3 Exists by ubernostrum on Friday December 18, 2015 @02:35PM (#10760453)
How do you know that? Any user supplied text may contain those things.

Which is why, in addition to things like length checks, we use other checks. But length is a quick and easy check for many types of values, and should be supported.

I've been down in the depths of Unicode behavior a time or two. I know what lurks down there, and I know from experience that your "strings should never allow these operations" stance is just as dumb as the more-common "I don't know what Unicode really is" stance. Can length checks and slicing and other operations run into trouble in some corners of Unicode? Sure. But if "this might cause trouble" were grounds for forbidding programmers ever to do something, we wouldn't have computers, any useful software at all, the internet or most of the other things we like and take for granted despite the fact that we occasionally have to fix bugs in it.

Now, get down off your snobby horse and join the rest of the real world, where we know that something might not be guaranteed reliable in all logically-possible cases, but find ways to work with it anyway because in 99.99% (or more) of the cases our code deals with it's good enough.

Re:Why Python 3 Exists by the_mitsuhiko on Thursday December 17, 2015 @01:49PM (#10753485)
To add to that: O(1) indexing into strings outside of the ASCII range is completely pointless and makes absolutely no sense. A language should disallow that and not encourage it.
Re:Why Python 3 Exists by Veedrac on Thursday December 17, 2015 @02:29PM (#10753798)
I disagree with it being pointless. Being able to get indices into a string for fast slicing is useful. Consider Regex where references get stored to slices in the array.

What you can drop is the idea of the index being an integer with any particular semantic meaning. Rust uses byte indexes with an assertion that the index lies on a codepoint boundary, for instance.

Re:Why Python 3 Exists by TheCoelacanth on Friday December 18, 2015 @01:42PM (#10760085)
That isn't an index into a string. It is an opaque indicator of position in a string. That opaque indicator of position happens to be implemented as an index into the underlying array, but it is not itself an index.
Re:Why Python 3 Exists by Veedrac on Saturday December 19, 2015 @01:21PM (#10764337)
That sounds like an index to me, and the operation surely indexing.

I mean, if one has a hashmap

    hash = {"foo": 1, "bar": 2}
we say hash["foo"] is an indexing operation, and "foo" is an index. Seems the same to me.
Re:Why Python 3 Exists by the_mitsuhiko on Thursday December 17, 2015 @03:51PM (#10754450)
> Being able to get indices into a string for fast slicing is useful.

Outside of the ASCII range? I doubt it. That most likely means you are doing something fundamentally wrong.

Re:Why Python 3 Exists by rbanffy on Thursday December 17, 2015 @05:41PM (#10755153)
Outside of the anglocentric world we also think in terms of characters and, yet, we consider ASCII as a rather primitive technology.
Re:Why Python 3 Exists by heinrich5991 on Thursday December 17, 2015 @05:03PM (#10754911)
I think something like a lexer doesn't need to know about graphemes, it just needs to know which codepoints are XID_start and XID_continue for identifiers and hardcoded codepoints for everything else.

EDIT: Or a JSON parser, etc.

Re:Why Python 3 Exists by the_mitsuhiko on Thursday December 17, 2015 @05:12PM (#10754967)
A lexer scans LTR. It never jumps codepoints ahead. Indexing in a lexer never makes sense.
Re:Why Python 3 Exists by tomp on Thursday December 17, 2015 @06:46PM (#10755574)
Obviously, it does; instead of allocating a new string for every token, you can just save the start & end index of a token, and actually allocate a new string only in the case where the parser actually requests/uses that token in the AST.
Re:Why Python 3 Exists by nostrademons on Thursday December 17, 2015 @09:16PM (#10756190)
But the type signature of that token should make it clear those are byte offsets and not character offsets. You can use them for byte-wise operations like memcpy, or to explicitly decode them into a string using the original encoding of the string. You cannot use them for string-wise operations like substring without explicitly casting or converting them to a string.

It seems like this whole mess comes from conflating bytes and characters. They are not the same, any more than integers and booleans are the same just because you can jnz in most assembly languages.

Re:Why Python 3 Exists by Veedrac on Friday December 18, 2015 @01:15AM (#10756969)
> You cannot use them for string-wise operations like substring without explicitly casting or converting them to a string.

Why on earth not? Any slice of a string (with a non-pathological internal encoding) with ends on codepoint boundaries is itself a valid string. Losing that information sounds like a whole lot of pointless hassle and potential for harm. It also forces the internal string encoding to be part of the interface, since taking a slice of a string and getting a byte array requires knowing how the string is encoded to use it, but getting another string means that such details are private.

It's true that they're byte offsets, but that doesn't matter. The parser doesn't care whether it's on the 10543th grapheme or the 10597th. It just wants to know where its stuff is.

Re:Why Python 3 Exists by nostrademons on Thursday December 17, 2015 @02:28PM (#10753788)
Swift has very sane way of doing this: Strings are sequences of Characters. Characters represent extended grapheme clusters. Iteration consumes as many bytes from the string as are necessary to read the next grapheme. In most cases, this is straight-up UTF-8 decoding, which can be implemented very efficiently; you only need to worry about extended grapheme clusters when you get to the non-ASCII subset of Unicode, so the code paths that require unicode table lookups are infrequently exercised.

String searching & matching return an opaque String.Index type. I assume that under the hood they do Boyer-Moore on bytes and String.Index is a byte offset, but the important thing is that String.Index values are not convertible to integers, and so you never run into the case where a user passes in a byte offset that would slice a grapheme in half. Instead, String.Index has properties for the next and previous index and a method to advance by an arbitrary amount, so you'd access the 7th character after a dash in a string as myString[myString.rangeOfString("-").advanceBy(7)]

Swift gives up on the idea that len(x+y) = len(x) + len(y). This is just something you have to remember; in UI programming, however, it makes a lot of sense because adding an accent to an existing string isn't going to make it take up more width in the text box. (x + y).utf8.count == x.utf8.count + y.utf8.count, however.

https://developer.apple.com/library/ios/documentation/Swift/...

If I were to adapt this to a domain like Python's, where string processing is pretty important, I'd allow indexing & slicing by integers (including negative integers), but I'd define them in terms of String.Index types, which iterate over extended graphemes under the hood:

  str[2] == str[str.start.advanceBy(2)]
  str[-3] == str[str.end.advanceBy(-3)]
  str[str.find('foo') + 3] == str[str.find('foo').advanceBy(3)]
  str[0:-3] == str.substring(str.start, str.end.advanceBy(-3))
  del str[0:3] == str.removeRange(str.start, str.start.advanceBy(3))
Re:Why Python 3 Exists by Veedrac on Thursday December 17, 2015 @02:32PM (#10753824)
> Swift gives up on the idea that len(x+y) = len(x) + len(y). This is just something you have to remember; in UI programming, however, it makes a lot of sense because adding an accent to an existing string isn't going to make it take up more width in the text box. (x + y).utf8.count == x.utf8.count + y.utf8.count, however.

So do fullwidth characters have a length of 2?

If not, how is this useful for finding the size of text in a text box anyway?

Re:Why Python 3 Exists by nostrademons on Thursday December 17, 2015 @02:46PM (#10753948)
I'm speaking more to the idea that when you concatenate strings, various important properties might not change. To get the actual size in pixels, you'd do str.sizeWithAttributes([NSFontAttributeName: myFont]).width, which is a whole other can of worms.
Re:Why Python 3 Exists by catnaroek on Thursday December 17, 2015 @08:05PM (#10755925)
It's possible to do better than existing mainstream solutions:

(0) Dependent types: The type of valid indices into a string depends, well, on the string. It's this type, not `uint`, that should be used by functions like `str.find()` and `re.match()`. This makes it a compile error to, say, slice a string with an index to another string. Internally, or course, all indices are represented as `uint`s, but your program has no business knowing this. It's an abstract type. IMO, this solution is particularly useful for systems languages, where the increased flexibility and fine-grained control are worth the increase in complexity.

(1) Coalgebras: `str.find()` shouldn't return an index. Instead, it should return in O(1) time, and allocating O(1) additional space, the left and right slices of the original string, split at the desired index. (If the string can be split in multiple places, `str.find()` should return an iterator whose element type is such left-right pairs.) For this to work, it's fundamental that the left and right slices not be allocated in their own buffers, but instead share the original string's buffer. IMO, this solution is best for high-level languages, where simplifying the API is well worth a little loss in flexibility.

Neither of these APIs allows the programmer to split strings at the wrong places.

Re:Why Python 3 Exists by lmm on Thursday December 17, 2015 @02:36PM (#10753864)
Python is an abstraction a long way above the Von Neumann machine. It should not expose implementation details like the number of bytes a string happens to be stored as. The whole point of the Python 2 -> 3 transition was to disallow just treating a string as a random pile of bytes and hoping it all works out.
Re:Why Python 3 Exists by hahainternet on Thursday December 17, 2015 @01:50PM (#10753497)
> Please, tell me what you actually think is correct behavior. It is far, far more useful than pointing out that something is "wrong" on purely semantic arguments.

You say "wrong" as if that's a quote from me. But it's not. Yes, strings have to be arrays, but how they work depends on the semantics of your language.

I'm not entirely happy with Python 3 because I don't think they really took their time to improve on the warts. I doubt there's a one-size-fits-all solution.

Re:Why Python 3 Exists by raiph on Friday December 18, 2015 @02:01PM (#10760220)
> I'm getting tired of this argument

Perhaps this is because Unicode is an exhaustive standard. :)

Seriously, I hear ya.

> every argument had the same hole that I see in your post: What, exactly, is a "character"? Please answer me that

I'd say the answer varies depending on who's talking, what they're talking about, and who's listening.

If it's a programmer interested in listening to the Unicode consortium and interested in what Unicode.org documents define for "what a user thinks of as a character" then the answer is, according to those documents, a "grapheme".

> then you can write a proposal for how the API should work, exactly

Right.

It can get, and has indeed gotten, better than that; one can write specifications, build reference implementations, and try things out in battle for a few years. Several text processing systems (including some programming languages) have gone this route and their results should be taken in to account. (ICU is perhaps the go to reference implementation.)

> and why this works with Indic and German and Korean and everything else.

For designs and implementations of text processing systems that claim an aspiration of progressing toward fully following the Unicode specification, the simplest answer to "why does it work?" (or "why it is expected to work") with Indic and German and Korean and everything else is of course the Unicode standard itself.

For those of us who aren't blessed with the patience of a saint and thus haven't read the entire spec from start to finish a few times, one has to rely to a degree on those who do, even if it's really hard to follow what the heck they're saying.

> You're trying to prevent people from slicing up grapheme clusters, which is a noble goal.

Given that grapheme clusters correspond to what the Unicode standard specifies for identifying and preserving the integrity of "what a user thinks of as a character", it seems less a noble goal than a fundamental eventual goal for any text processing system that explicitly claims to aspire to support the world's human languages via Unicode compliance.

> we don't end up slicing grapheme clusters very often in practice.

Which "we" are you referring to?

The western world has dominated text processing to date does. This must not mean that "we", by which I mean all humans, would best continue practices that marginalize those with native languages not well served. (By ignoring the central importance of "what a user thinks of as a character"; again, I'm quoting Unicode consortium documentation.)

Or, more parochially, what about the "we" that's just programmers who want to be able to increase their confidence that they are properly processing Unicode compliant text?

> In the meantime, formats like JSON and XML are defined in terms of code points ...

Code points are the middle level of Unicode, sandwiched between byte encodings at the bottom level and graphemes at the top. There's nothing wrong with a text processing system stopping at the middle level provided folk don't lose sight of the fact that it stops at the middle level.

Thinking that code points are "what a user thinks of as a character" mostly works out OK when processing text that's English or the like, but is definitively not Unicode compliant to the degree it's used too broadly.

> Can you tell me what the correct behavior is?

I would suggest that the most appropriate authority is the Unicode consortium and its Unicode standard.

> Might you end up with bugs in programs because len(x + y) != len(x) + len(y)?

Oh for sure. But it's part of the Unicode standard. It's one of several apparently strange discontinuities that the Unicode effort introduced last century in an attempt to deal with the combination of human language reality and computer system realities. They may have chosen the wrong sweetspot. But now Unicode is a significant part of reality. I, for one, don't think it's wise or even realistic to abandon the Unicode standard as it is...