News for hackers. Stuff that matters.
Re:Why Python 3 Exists by dietrichepp on Thursday December 17, 2015 @12:09PM (#10752669)
Hm, I never saw this as ambiguous at all, except for a few weird encodings that Python has as "convenience" method.

Here's how you remember it: "Unicode" is not an encoding. It never was, it never will be. Of course, the data must be encoded in memory somehow, but in Python 3, you cannot be sure what encoding that is because it's not really exposed to the user. From what I understand, there are different encodings that string objects will use, transparently, in order to save memory!

You always "encode" something into bytes, and "decode" bytes back into something. There should be exactly two functions, because the functions have different types: "encode" is str -> bytes, "decode" is bytes -> str. Explicit is better than implicit.

    output = input.decode(self.coding)
With Python 3, I instantly know that "input" is bytes and "output" is str.
Re:Why Python 3 Exists by cgrubb on Thursday December 17, 2015 @04:55PM (#10754851)
> You always "encode" something into bytes, and "decode" bytes back into something. There should be exactly two functions, because the functions have different types: "encode" is str -> bytes, "decode" is bytes -> str. Explicit is better than implicit.

I'm not sure "encoding" tells us that the output is bytes. Surely information can be encoded in other ways. Names like .to_bytes() and .from_bytes() might have saved a lot of trips to the documentation.

Re:Why Python 3 Exists by mark-r on Thursday December 17, 2015 @11:05PM (#10756560)
I would use .to_str() instead of .from_bytes(), since the action is from the perspective of the object you're calling it on.
Re:Why Python 3 Exists by viraptor on Friday December 18, 2015 @12:16AM (#10756780)
> Surely information can be encoded in other ways.

Information - yes. Strings... I'm not sure. What other representations are you thinking about?

Re:Why Python 3 Exists by msellout on Friday December 18, 2015 @01:47AM (#10757071)
urlencode(mapping) --> querystring
Re:Why Python 3 Exists by twoodfin on Thursday December 17, 2015 @01:22PM (#10753240)
Indeed, the Python 3 Unicode string object is fascinatingly clever. Code worth reading:

https://github.com/python/cpython/blob/master/Objects/unicod...

Re:Why Python 3 Exists by nostrademons on Thursday December 17, 2015 @01:57PM (#10753571)
Also, it's incompatible with UTF-8 strings stored in C, which means that when you cross the Python/C API boundary, you have to re-encode all strings. This is a large performance penalty right at the time when you can least afford performance penalties.

IMNSHO, most modern languages should be storing strings as UTF-8 and give up on random access by characters. You almost never need it; in the most frequent case where you do (using indexOf or equivalent to search for a substring, and then breaking on it), you can solve the problem by returning a type-safe iterator or index object that contains a byte offset under the hood, and then slicing on that. Go, Rust, and Swift have all gone this route.

Re:Why Python 3 Exists by viraptor on Friday December 18, 2015 @12:31AM (#10756835)
> Also, it's incompatible with UTF-8 strings stored in C

That's not true as of 3.3. It will store the compact representation, which in most cases is ASCII, which is UTF-8 compatible. https://www.python.org/dev/peps/pep-0393/

So unless you go outside of UCS-1, you don't need to reencode anything.

Re:Why Python 3 Exists by shoyer on Thursday December 17, 2015 @02:29PM (#10753802)
This design doc from DyND (a possible NumPy alternative) has some useful references on this point: https://github.com/libdynd/libdynd/blob/master/docs/string-d...
Re:Why Python 3 Exists by rurban on Thursday December 17, 2015 @06:54PM (#10755622)
Nope. This is just a normal short-string implementation, inlined with 64bit systems, with a tag bit. Very common on every better VM.

This has nothing to do with (inefficient) encodings, and their bad historic namings, probably derived from perl.

Of course any name like byte2utf8 or just to_byte or b2u8 would have been better than encode/decode.

And of course cache size matters nowadays much more than immediate substring access for utf8, so nobody should use ucs-2 or even ucs-4 internally. This is easily benchmarkable.

Re:Why Python 3 Exists by 0x0 on Thursday December 17, 2015 @08:34PM (#10756032)
"Byte2utf8" is a pretty confusing name for a method, considering utf8 is a byte encoding of unicode... :)
Re:Why Python 3 Exists by Matthias247 on Friday December 18, 2015 @03:53AM (#10757435)
However C also did not agree on UTF8. There are lots libraries which use UTF16 encoding, so you would need to convert for these if you stick to UTF8. And you maybe need some conversion anyway, because one system expects zero terminated strings while another expects explicit length fields. Unfortunately strings are really a mess.
Re:Why Python 3 Exists by the_mitsuhiko on Thursday December 17, 2015 @01:48PM (#10753472)
> Indeed, the Python 3 Unicode string object is fascinatingly clever. Code worth reading:

Clever is relative. The Python 3 unicode string objects makes very little sense given real world situations. It wastes enormous amount of memory in many common scenarios for no reason at all other than to facilitate O(1) access to charpoints which is completely inappropriate for text processing anyways.

Worse than that: it can blow up by another 50% the moment someone encodings it to utf-8 as the encoded version is cached around.

The object is too clever for its own good.

Re:Why Python 3 Exists by klodolph on Thursday December 17, 2015 @02:43PM (#10753920)
You're right that O(1) access is pointless, and UTF-8 is fine. But there are space optimizations on the Python string object which make it so it doesn't waste that much space. Basically, it dynamically selects LATIN-1, UCS-2, or UTF-32 (or you could call them UCS-1, UCS-2, UCS-4, but there isn't really any such thing as UCS-1, and UCS-4 is an obsolete name for UTF-32).

    >>> import sys
    >>> def charsize(x):
    ...     return sys.getsizeof(x + x) - sys.getsizeof(x)
    ...
    >>> charsize(b'A')
    1
    >>> charsize('A')
    1
    >>> charsize('\xff')
    1
    >>> charsize('\uffff')
    2
    >>> charsize('\U0010FFFF')
    4
Re:Why Python 3 Exists by the_mitsuhiko on Thursday December 17, 2015 @03:51PM (#10754442)
> But there are space optimizations on the Python string object which make it so it doesn't waste that much space

Except it does not help. When you render out a template it starts out as latin1, then the first time it hits a unicode char the whole thing re-encodes into ucs2 and then when it sees an emoji it re-encodes to ucs4. In practical terms you will see UCS2 and UCS4 all the time.

Re:Why Python 3 Exists by aidenn0 on Tuesday December 22, 2015 @07:03PM (#10781192)
Except UTF-8 is never less compact than UTF-32 (it is one to four octets, and UTF-32 is always four octets), and is often more compact than UTF-16, with the sole exception being BMP CJK text that is not embedded in an ascii-based markup language.
Re:Why Python 3 Exists by Animats on Friday December 18, 2015 @12:34AM (#10756843)
Another alternative would have been to store strings in UTF-8 and generate an array of indices to the starting point of each entity only if needed for random access. Subscripting a string would use the array of indices.

Only a few standard member functions of string (find, rfind, index, rindex) return integer subscripts into a string. Those could return an opaque index object which has an index and a link to the string. Such objects could be used for string operations which need a string index. If an index object is forcibly converted to an integer, the array of indices has to be generated, but otherwise, it is unnecessary. You could even allow adding or subtracting integers from an index object, which would walk the UTF-8 string forwards or backwards as indicated.

This would make UTF-8 strings usable with subscript and slice notation without exploding them to wide characters.

Re:Why Python 3 Exists by eugenekolo2 on Thursday December 17, 2015 @11:40PM (#10756682)
Yeah, I'm not reading 15K LOC... What makes it clever? What is even the point of linking a 15K file?
Re:Why Python 3 Exists by est on Thursday December 17, 2015 @09:15PM (#10756184)
> "Unicode" is not an encoding.

It's important to point out that "Unicode" in Windows is an encoding. It means specifically UTF-16 LE, many people like me learned the concept "Unicode" on Windows got confused to hell when learning Python.

Re:Why Python 3 Exists by mark-r on Thursday December 17, 2015 @11:21PM (#10756604)
Microsoft - I don't know whether to hate on them or pity them. They were an early adopter of Unicode, back when there was only one encoding that probably didn't even have a name yet (UCS-2). It's easy to see how they would unleash that confusing nomenclature on the world, but they should have made an attempt to clear it up when other encodings became available, and they didn't.
Re:Why Python 3 Exists by agumonkey on Thursday December 17, 2015 @01:37PM (#10753371)
It is ambiguous. When you still don't get encodings. As surprising as this can be, I click faster on monads than encodings (maybe it doesn't tease my brain enough for me to retain it).
Re:Why Python 3 Exists by aidenn0 on Tuesday December 22, 2015 @06:59PM (#10781176)
It is not true that unicode never was an encoding. It was originally a fixed-width 16-bit encoding[1]. This was arguably no longer true in the early '90s and definitely not true by 1996.

1: http://www.unicode.org/history/unicode88.pdf

Re:Why Python 3 Exists by njharman on Thursday December 17, 2015 @07:37PM (#10755825)
ok, but what I read from a file is that bytes or "something".

When save some "things" I read from utf16 file and want to save them to Postgresql who's client encoding is set to utf8, what do I do?

Re:Why Python 3 Exists by hahainternet on Thursday December 17, 2015 @12:17PM (#10752727)

    In [1]: len("नि")
    Out[1]: 2

So close, but it's not really a string or bytes, but a codepoint array. You can even iterate it:

    In [2]: for c in "नि":
       ...:     print("Character {}".format(c))
       ...:     
    Character न
    Character ि
Re:Why Python 3 Exists by dietrichepp on Thursday December 17, 2015 @12:47PM (#10752971)
I'm getting tired of this argument, we had it in the Rust community as well and I was tired of it there, too. I've seen the same argument over and over again, and every argument had the same hole that I see in your post: What, exactly, is a "character"? Please answer me that, and then you can write a proposal for how the API should work, exactly, and why this works with Indic and German and Korean and everything else.

Strings have to be arrays. It is inevitable. We are on Von Neumann machines with L1 cache lines that are something like 64 bytes wide, so making strings into something other than arrays is a complete non-starter. Because strings are arrays, we use array indexes to slice strings. The indexes are integers, because that just makes sense on a Von Neumann machine.

So your complaint is that you can slice a string in "bad" ways. What makes it bad? You're trying to prevent people from slicing up grapheme clusters, which is a noble goal. But in practice, we get the indexes from methods like str.find() or re.match(), and we treat them as more or less opaque, so we don't end up slicing grapheme clusters very often in practice. Grapheme cluster slicing requires a big Unicode table in memory anyway, so in the rare case that you need it, you can pay the high performance cost for using it. In the meantime, formats like JSON and XML are defined in terms of code points, so using code point arrays eliminates a class of bugs where you could accidentally make malformed XML or JSON, which would then get completely REJECTED by the receiving side, causing your Jabber client to quit or your web browser to show a bunch of mojibake.

And let me ask you this: what do I get when I write:

    x = "a"
    y = "\u0301"
Is the resulting len(x + y) == 1? But len(x) == 1 and len(y) == 1? Can you tell me what the correct behavior is? Might you end up with bugs in programs because len(x + y) != len(x) + len(y)? Or do you introduce extra code points into the string when you concatenate them?

Please, tell me what you actually think is correct behavior. It is far, far more useful than pointing out that something is "wrong" on purely semantic arguments.

Re:Why Python 3 Exists by lucozade on Thursday December 17, 2015 @02:06PM (#10753631)
> Because strings are arrays, we use array indexes to slice strings

But that's a choice and not the only one. Sure you're likely to implement your Unicode string as an array of some type. But that doesn't mean that the only sensible approach is to expose that array directly to the user. Or more specifically as the primary interface.

For my money, Swift has probably the most comprehensive Unicode string API I've seen [0]. What they do is, essentially, have an opaque String type but support various "views" as properties. The main view is called .characters and represents a collection of grapheme clusters. They also have properties that present Unicode scalars, utf8 encoding etc.

Their API is complex, no question. But then proper handling of Unicode is complex. But it does show that there are other options than simply exposing the Unicode scalars.

BTW to your point on len(x) + len(y). The answer is 2 if you define len() on Unicode scalars but 1 if you define it on characters. Why? because len("\u301") should be 0. It's not a character, it's a base modifier. It is, of course, true that getting this right is likely to be substantially more expensive than getting it wrong but that doesn't mean it can't be done.

[0] https://developer.apple.com/library/ios/documentation/Swift/...

Re:Why Python 3 Exists by snuxoll on Thursday December 17, 2015 @06:45PM (#10755568)
Check out Perl 6's strings, they are implemented similarly to the Swift usage of string "views" but with a default that assumes you want NFC with the ability to chose alternate normalizations and raw access to codepoints if you want that instead of grapheme clusters.
Re:Why Python 3 Exists by raiph on Friday December 18, 2015 @12:05PM (#10759468)
> Swift has probably the most comprehensive Unicode string API I've seen

Perl 6 has long specified an interesting and rich approach to Unicode. Have you explored it?

(An initial implementation for an initial release of the Rakudo compiler was worked on over the last few years and declared done, modulo bugs, a few weeks ago. A recent blog post provides a friendly overview.[1].)

One notable difference is that Swift adopts an iteration-only view of strings for dealing with characters whereas Perl 6 does not. One consequence is that indexing operations in Perl 6 are O(1) (time) at the expense of potentially (and for the current implementation, typically) greater use of RAM.

> BTW to your point on len(x) + len(y). The answer is 2 if you define len() on Unicode scalars but 1 if you define it on characters. Why? because len("\u301") should be 0.

Are you 100% sure of this of your last sentence ("should be 0")? I get that it's intuitively reasonable that an isolated base modifier be length 0 but I thought the Unicode standard specified (or at least recommended) that it be 1, and I note that Perl 6 treats isolated base modifiers (or at least this particular one) as length 1:

> ("a").chars 1

> (0x301.chr).chars 1

> ("a" ~ 0x301.chr).chars 1

So both Swift and Perl 6 correctly report the length of "a" as 1 and an "a" concatenated with "\u301" as 1 but disagree on what to do with an isolated base modifier.

> It is, of course, true that getting this right is likely to be substantially more expensive than getting it wrong but that doesn't mean it can't be done.

Indeed.

[1] https://perl6advent.wordpress.com/2015/12/07/day-7-unicode-p...

Re:Why Python 3 Exists by klodolph on Thursday December 17, 2015 @02:45PM (#10753935)
len("\u0301") == 0 is extremely surprising.
Re:Why Python 3 Exists by tomp on Thursday December 17, 2015 @06:48PM (#10755591)
Not really. "\u0301".characters should be equivalent to "".characters, so len("\u0301".characters) == 0 == len("".characters) makes sense.
Re:Why Python 3 Exists by lucozade on Thursday December 17, 2015 @03:53PM (#10754464)
Of course. What you were expecting was 6 right?

You learnt ASCII. Then you learnt about escaping to get around limitations in ASCII. Then you learnt that Unicode got around non-Latin issues with ASCII.

Each step has added cognitive load. It seems surprising to me that the next step wouldn't.

Re:Why Python 3 Exists by klodolph on Thursday December 17, 2015 @04:13PM (#10754587)
The surprise comes with a cost, and the cost is in bugs. The point of Python 3 is to eliminate some bugs in Python 2, and forcing developers to redesign their code around a completely new string API--no matter if that new API is more semantically sound--will come with its own cost of introduced bugs and more painful conversions, which goes against the purpose of Python 3.

I'm not saying that the string API in Python is perfection, but it had a specific role to fill in the larger Python ecosystem, and it does that very well.

Anyway... I would say that len(x) == 0 should be the same as saying that x is empty. According to the Unicode standard, the number of grapheme clusters in "\u0301" is 1, anyway! And on most systems, the string will display as an acute accent in its own space, and you can place the cursor before or after it.

Not like the set of valid "cursor positions" in a string isn't a locale-dependent value, anyway. Grapheme clusters can be thought of to be locale-independent but they are NOT the same as cursor positions.

Re:Why Python 3 Exists by lucozade on Thursday December 17, 2015 @04:53PM (#10754843)
Not to sound too factious but the decision to redesign the string API was a big reason why Python 3 has been a little on the slow cooked side. In for a penny and all that.

If I recall correctly the modifier on it's own isn't correct. There is a base that should go before it to produce a valid character. I don't recall the base unfortunately and I don't have the spec to hand.

Re:Why Python 3 Exists by klodolph on Friday December 18, 2015 @12:00PM (#10759444)
What do you mean by "correct"? The Unicode standard talks about well-formed UTF-8/16/32, but in general, any sequence of code points is permissible. By derivation, any sequence of code points can appear in JSON strings, and any sequence of code points excluding control characters can appear in XML. I personally have read the Unicode spec, and used it to implement some of the algorithms presented. This includes the grapheme cluster breaking algorithm... the case of combining characters at the beginning of a line is, in fact, covered by the spec, and cause a break both before and afterwards. In general, there are other cases where grapheme cluster breaks are not preserved by string concatenation, and you often want a locale-specific version anyway.
Re:Why Python 3 Exists by lucozade on Friday December 18, 2015 @01:13PM (#10759891)
> What do you mean by "correct"?

I mean this from the standard: "Nonspacing combining marks used by the Unicode Standard may be exhibited in apparent isolation by applying them to U+00A0 no-break space. "

You can have valid Unicode that doesn't do this. But it's not intended to represent the character as some kind of textual thing. It's intended so that you can build up Unicode strings sensibly. Same idea as allowing mis-matched surrogates.

Re:Why Python 3 Exists by anon1385 on Thursday December 17, 2015 @01:02PM (#10753088)
The correct answer is that strings shouldn't have length methods. There is no such thing as the 'length' of a string because as you pointed out there is no clear definition of what a 'character' is.

https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-s...

Re:Why Python 3 Exists by ubernostrum on Thursday December 17, 2015 @05:20PM (#10755011)
The correct answer is that strings shouldn't have length methods

There are plenty of uses for getting the length of a string where you will not be running into combining characters, clusters or any other things which would require special handling.

(remember that on the internet, all the data you receive is initially in string form, and very often we want to receive data representing fixed-length values such as phone numbers or credit card numbers, so "you're wrong for wanting to know the length" is both a non-starter and, well, just plain wrong)

Re:Why Python 3 Exists by anon1385 on Friday December 18, 2015 @05:13AM (#10757678)
>There are plenty of uses for getting the length of a string where you will not be running into combining characters, clusters or any other things which would require special handling.

How do you know that? Any user supplied text may contain those things.

>remember that on the internet, all the data you receive is initially in string form

It's not. It's bytes.

>very often we want to receive data representing fixed-length values such as phone numbers or credit card numbers

This is a good example of why strings shouldn't have a length property - people don't understand what length actually does in most languages. If you think that a len() function can verify that the contents of a text field is in the format of a CC number then you don't understand the len function. It can't do what you are asking. See the examples given in the GP (len("\u0301") for example).

The fact that the people who advocate for length functions want to use them for things that they are broken for is exactly why length functions shouldn't exist.

Re:Why Python 3 Exists by ubernostrum on Friday December 18, 2015 @02:35PM (#10760453)
How do you know that? Any user supplied text may contain those things.

Which is why, in addition to things like length checks, we use other checks. But length is a quick and easy check for many types of values, and should be supported.

I've been down in the depths of Unicode behavior a time or two. I know what lurks down there, and I know from experience that your "strings should never allow these operations" stance is just as dumb as the more-common "I don't know what Unicode really is" stance. Can length checks and slicing and other operations run into trouble in some corners of Unicode? Sure. But if "this might cause trouble" were grounds for forbidding programmers ever to do something, we wouldn't have computers, any useful software at all, the internet or most of the other things we like and take for granted despite the fact that we occasionally have to fix bugs in it.

Now, get down off your snobby horse and join the rest of the real world, where we know that something might not be guaranteed reliable in all logically-possible cases, but find ways to work with it anyway because in 99.99% (or more) of the cases our code deals with it's good enough.

Re:Why Python 3 Exists by the_mitsuhiko on Thursday December 17, 2015 @01:49PM (#10753485)
To add to that: O(1) indexing into strings outside of the ASCII range is completely pointless and makes absolutely no sense. A language should disallow that and not encourage it.
Re:Why Python 3 Exists by Veedrac on Thursday December 17, 2015 @02:29PM (#10753798)
I disagree with it being pointless. Being able to get indices into a string for fast slicing is useful. Consider Regex where references get stored to slices in the array.

What you can drop is the idea of the index being an integer with any particular semantic meaning. Rust uses byte indexes with an assertion that the index lies on a codepoint boundary, for instance.

Re:Why Python 3 Exists by TheCoelacanth on Friday December 18, 2015 @01:42PM (#10760085)
That isn't an index into a string. It is an opaque indicator of position in a string. That opaque indicator of position happens to be implemented as an index into the underlying array, but it is not itself an index.
Re:Why Python 3 Exists by Veedrac on Saturday December 19, 2015 @01:21PM (#10764337)
That sounds like an index to me, and the operation surely indexing.

I mean, if one has a hashmap

    hash = {"foo": 1, "bar": 2}
we say hash["foo"] is an indexing operation, and "foo" is an index. Seems the same to me.
Re:Why Python 3 Exists by the_mitsuhiko on Thursday December 17, 2015 @03:51PM (#10754450)
> Being able to get indices into a string for fast slicing is useful.

Outside of the ASCII range? I doubt it. That most likely means you are doing something fundamentally wrong.

Re:Why Python 3 Exists by rbanffy on Thursday December 17, 2015 @05:41PM (#10755153)
Outside of the anglocentric world we also think in terms of characters and, yet, we consider ASCII as a rather primitive technology.
Re:Why Python 3 Exists by heinrich5991 on Thursday December 17, 2015 @05:03PM (#10754911)
I think something like a lexer doesn't need to know about graphemes, it just needs to know which codepoints are XID_start and XID_continue for identifiers and hardcoded codepoints for everything else.

EDIT: Or a JSON parser, etc.

Re:Why Python 3 Exists by the_mitsuhiko on Thursday December 17, 2015 @05:12PM (#10754967)
A lexer scans LTR. It never jumps codepoints ahead. Indexing in a lexer never makes sense.
Re:Why Python 3 Exists by tomp on Thursday December 17, 2015 @06:46PM (#10755574)
Obviously, it does; instead of allocating a new string for every token, you can just save the start & end index of a token, and actually allocate a new string only in the case where the parser actually requests/uses that token in the AST.
Re:Why Python 3 Exists by nostrademons on Thursday December 17, 2015 @09:16PM (#10756190)
But the type signature of that token should make it clear those are byte offsets and not character offsets. You can use them for byte-wise operations like memcpy, or to explicitly decode them into a string using the original encoding of the string. You cannot use them for string-wise operations like substring without explicitly casting or converting them to a string.

It seems like this whole mess comes from conflating bytes and characters. They are not the same, any more than integers and booleans are the same just because you can jnz in most assembly languages.

Re:Why Python 3 Exists by Veedrac on Friday December 18, 2015 @01:15AM (#10756969)
> You cannot use them for string-wise operations like substring without explicitly casting or converting them to a string.

Why on earth not? Any slice of a string (with a non-pathological internal encoding) with ends on codepoint boundaries is itself a valid string. Losing that information sounds like a whole lot of pointless hassle and potential for harm. It also forces the internal string encoding to be part of the interface, since taking a slice of a string and getting a byte array requires knowing how the string is encoded to use it, but getting another string means that such details are private.

It's true that they're byte offsets, but that doesn't matter. The parser doesn't care whether it's on the 10543th grapheme or the 10597th. It just wants to know where its stuff is.

Re:Why Python 3 Exists by nostrademons on Thursday December 17, 2015 @02:28PM (#10753788)
Swift has very sane way of doing this: Strings are sequences of Characters. Characters represent extended grapheme clusters. Iteration consumes as many bytes from the string as are necessary to read the next grapheme. In most cases, this is straight-up UTF-8 decoding, which can be implemented very efficiently; you only need to worry about extended grapheme clusters when you get to the non-ASCII subset of Unicode, so the code paths that require unicode table lookups are infrequently exercised.

String searching & matching return an opaque String.Index type. I assume that under the hood they do Boyer-Moore on bytes and String.Index is a byte offset, but the important thing is that String.Index values are not convertible to integers, and so you never run into the case where a user passes in a byte offset that would slice a grapheme in half. Instead, String.Index has properties for the next and previous index and a method to advance by an arbitrary amount, so you'd access the 7th character after a dash in a string as myString[myString.rangeOfString("-").advanceBy(7)]

Swift gives up on the idea that len(x+y) = len(x) + len(y). This is just something you have to remember; in UI programming, however, it makes a lot of sense because adding an accent to an existing string isn't going to make it take up more width in the text box. (x + y).utf8.count == x.utf8.count + y.utf8.count, however.

https://developer.apple.com/library/ios/documentation/Swift/...

If I were to adapt this to a domain like Python's, where string processing is pretty important, I'd allow indexing & slicing by integers (including negative integers), but I'd define them in terms of String.Index types, which iterate over extended graphemes under the hood:

  str[2] == str[str.start.advanceBy(2)]
  str[-3] == str[str.end.advanceBy(-3)]
  str[str.find('foo') + 3] == str[str.find('foo').advanceBy(3)]
  str[0:-3] == str.substring(str.start, str.end.advanceBy(-3))
  del str[0:3] == str.removeRange(str.start, str.start.advanceBy(3))
Re:Why Python 3 Exists by Veedrac on Thursday December 17, 2015 @02:32PM (#10753824)
> Swift gives up on the idea that len(x+y) = len(x) + len(y). This is just something you have to remember; in UI programming, however, it makes a lot of sense because adding an accent to an existing string isn't going to make it take up more width in the text box. (x + y).utf8.count == x.utf8.count + y.utf8.count, however.

So do fullwidth characters have a length of 2?

If not, how is this useful for finding the size of text in a text box anyway?

Re:Why Python 3 Exists by nostrademons on Thursday December 17, 2015 @02:46PM (#10753948)
I'm speaking more to the idea that when you concatenate strings, various important properties might not change. To get the actual size in pixels, you'd do str.sizeWithAttributes([NSFontAttributeName: myFont]).width, which is a whole other can of worms.
Re:Why Python 3 Exists by catnaroek on Thursday December 17, 2015 @08:05PM (#10755925)
It's possible to do better than existing mainstream solutions:

(0) Dependent types: The type of valid indices into a string depends, well, on the string. It's this type, not `uint`, that should be used by functions like `str.find()` and `re.match()`. This makes it a compile error to, say, slice a string with an index to another string. Internally, or course, all indices are represented as `uint`s, but your program has no business knowing this. It's an abstract type. IMO, this solution is particularly useful for systems languages, where the increased flexibility and fine-grained control are worth the increase in complexity.

(1) Coalgebras: `str.find()` shouldn't return an index. Instead, it should return in O(1) time, and allocating O(1) additional space, the left and right slices of the original string, split at the desired index. (If the string can be split in multiple places, `str.find()` should return an iterator whose element type is such left-right pairs.) For this to work, it's fundamental that the left and right slices not be allocated in their own buffers, but instead share the original string's buffer. IMO, this solution is best for high-level languages, where simplifying the API is well worth a little loss in flexibility.

Neither of these APIs allows the programmer to split strings at the wrong places.

Re:Why Python 3 Exists by lmm on Thursday December 17, 2015 @02:36PM (#10753864)
Python is an abstraction a long way above the Von Neumann machine. It should not expose implementation details like the number of bytes a string happens to be stored as. The whole point of the Python 2 -> 3 transition was to disallow just treating a string as a random pile of bytes and hoping it all works out.
Re:Why Python 3 Exists by hahainternet on Thursday December 17, 2015 @01:50PM (#10753497)
> Please, tell me what you actually think is correct behavior. It is far, far more useful than pointing out that something is "wrong" on purely semantic arguments.

You say "wrong" as if that's a quote from me. But it's not. Yes, strings have to be arrays, but how they work depends on the semantics of your language.

I'm not entirely happy with Python 3 because I don't think they really took their time to improve on the warts. I doubt there's a one-size-fits-all solution.

Re:Why Python 3 Exists by raiph on Friday December 18, 2015 @02:01PM (#10760220)
> I'm getting tired of this argument

Perhaps this is because Unicode is an exhaustive standard. :)

Seriously, I hear ya.

> every argument had the same hole that I see in your post: What, exactly, is a "character"? Please answer me that

I'd say the answer varies depending on who's talking, what they're talking about, and who's listening.

If it's a programmer interested in listening to the Unicode consortium and interested in what Unicode.org documents define for "what a user thinks of as a character" then the answer is, according to those documents, a "grapheme".

> then you can write a proposal for how the API should work, exactly

Right.

It can get, and has indeed gotten, better than that; one can write specifications, build reference implementations, and try things out in battle for a few years. Several text processing systems (including some programming languages) have gone this route and their results should be taken in to account. (ICU is perhaps the go to reference implementation.)

> and why this works with Indic and German and Korean and everything else.

For designs and implementations of text processing systems that claim an aspiration of progressing toward fully following the Unicode specification, the simplest answer to "why does it work?" (or "why it is expected to work") with Indic and German and Korean and everything else is of course the Unicode standard itself.

For those of us who aren't blessed with the patience of a saint and thus haven't read the entire spec from start to finish a few times, one has to rely to a degree on those who do, even if it's really hard to follow what the heck they're saying.

> You're trying to prevent people from slicing up grapheme clusters, which is a noble goal.

Given that grapheme clusters correspond to what the Unicode standard specifies for identifying and preserving the integrity of "what a user thinks of as a character", it seems less a noble goal than a fundamental eventual goal for any text processing system that explicitly claims to aspire to support the world's human languages via Unicode compliance.

> we don't end up slicing grapheme clusters very often in practice.

Which "we" are you referring to?

The western world has dominated text processing to date does. This must not mean that "we", by which I mean all humans, would best continue practices that marginalize those with native languages not well served. (By ignoring the central importance of "what a user thinks of as a character"; again, I'm quoting Unicode consortium documentation.)

Or, more parochially, what about the "we" that's just programmers who want to be able to increase their confidence that they are properly processing Unicode compliant text?

> In the meantime, formats like JSON and XML are defined in terms of code points ...

Code points are the middle level of Unicode, sandwiched between byte encodings at the bottom level and graphemes at the top. There's nothing wrong with a text processing system stopping at the middle level provided folk don't lose sight of the fact that it stops at the middle level.

Thinking that code points are "what a user thinks of as a character" mostly works out OK when processing text that's English or the like, but is definitively not Unicode compliant to the degree it's used too broadly.

> Can you tell me what the correct behavior is?

I would suggest that the most appropriate authority is the Unicode consortium and its Unicode standard.

> Might you end up with bugs in programs because len(x + y) != len(x) + len(y)?

Oh for sure. But it's part of the Unicode standard. It's one of several apparently strange discontinuities that the Unicode effort introduced last century in an attempt to deal with the combination of human language reality and computer system realities. They may have chosen the wrong sweetspot. But now Unicode is a significant part of reality. I, for one, don't think it's wise or even realistic to abandon the Unicode standard as it is...

Re:Why Python 3 Exists by Veedrac on Thursday December 17, 2015 @12:29PM (#10752831)
Although true, the aim of the transition was to enforce encoding-correctness. Operations on strs should never break the encoding itself[1], but preserving the semantic meaning of the text itself is basically impossible for arbitrary data.

[1] Sadly surrogate escapes break this. Rust's division into String:OsString:Vec<u8> makes that aspect a lot cleaner.