News for hackers. Stuff that matters.
Re:Why Python 3 Exists by anon1385 on Thursday December 17, 2015 @01:02PM (#10753088)
The correct answer is that strings shouldn't have length methods. There is no such thing as the 'length' of a string because as you pointed out there is no clear definition of what a 'character' is.

https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-s...

Re:Why Python 3 Exists by ubernostrum on Thursday December 17, 2015 @05:20PM (#10755011)
The correct answer is that strings shouldn't have length methods

There are plenty of uses for getting the length of a string where you will not be running into combining characters, clusters or any other things which would require special handling.

(remember that on the internet, all the data you receive is initially in string form, and very often we want to receive data representing fixed-length values such as phone numbers or credit card numbers, so "you're wrong for wanting to know the length" is both a non-starter and, well, just plain wrong)

Re:Why Python 3 Exists by anon1385 on Friday December 18, 2015 @05:13AM (#10757678)
>There are plenty of uses for getting the length of a string where you will not be running into combining characters, clusters or any other things which would require special handling.

How do you know that? Any user supplied text may contain those things.

>remember that on the internet, all the data you receive is initially in string form

It's not. It's bytes.

>very often we want to receive data representing fixed-length values such as phone numbers or credit card numbers

This is a good example of why strings shouldn't have a length property - people don't understand what length actually does in most languages. If you think that a len() function can verify that the contents of a text field is in the format of a CC number then you don't understand the len function. It can't do what you are asking. See the examples given in the GP (len("\u0301") for example).

The fact that the people who advocate for length functions want to use them for things that they are broken for is exactly why length functions shouldn't exist.

Re:Why Python 3 Exists by ubernostrum on Friday December 18, 2015 @02:35PM (#10760453)
How do you know that? Any user supplied text may contain those things.

Which is why, in addition to things like length checks, we use other checks. But length is a quick and easy check for many types of values, and should be supported.

I've been down in the depths of Unicode behavior a time or two. I know what lurks down there, and I know from experience that your "strings should never allow these operations" stance is just as dumb as the more-common "I don't know what Unicode really is" stance. Can length checks and slicing and other operations run into trouble in some corners of Unicode? Sure. But if "this might cause trouble" were grounds for forbidding programmers ever to do something, we wouldn't have computers, any useful software at all, the internet or most of the other things we like and take for granted despite the fact that we occasionally have to fix bugs in it.

Now, get down off your snobby horse and join the rest of the real world, where we know that something might not be guaranteed reliable in all logically-possible cases, but find ways to work with it anyway because in 99.99% (or more) of the cases our code deals with it's good enough.

Re:Why Python 3 Exists by the_mitsuhiko on Thursday December 17, 2015 @01:49PM (#10753485)
To add to that: O(1) indexing into strings outside of the ASCII range is completely pointless and makes absolutely no sense. A language should disallow that and not encourage it.
Re:Why Python 3 Exists by Veedrac on Thursday December 17, 2015 @02:29PM (#10753798)
I disagree with it being pointless. Being able to get indices into a string for fast slicing is useful. Consider Regex where references get stored to slices in the array.

What you can drop is the idea of the index being an integer with any particular semantic meaning. Rust uses byte indexes with an assertion that the index lies on a codepoint boundary, for instance.

Re:Why Python 3 Exists by TheCoelacanth on Friday December 18, 2015 @01:42PM (#10760085)
That isn't an index into a string. It is an opaque indicator of position in a string. That opaque indicator of position happens to be implemented as an index into the underlying array, but it is not itself an index.
Re:Why Python 3 Exists by Veedrac on Saturday December 19, 2015 @01:21PM (#10764337)
That sounds like an index to me, and the operation surely indexing.

I mean, if one has a hashmap

    hash = {"foo": 1, "bar": 2}
we say hash["foo"] is an indexing operation, and "foo" is an index. Seems the same to me.
Re:Why Python 3 Exists by the_mitsuhiko on Thursday December 17, 2015 @03:51PM (#10754450)
> Being able to get indices into a string for fast slicing is useful.

Outside of the ASCII range? I doubt it. That most likely means you are doing something fundamentally wrong.

Re:Why Python 3 Exists by rbanffy on Thursday December 17, 2015 @05:41PM (#10755153)
Outside of the anglocentric world we also think in terms of characters and, yet, we consider ASCII as a rather primitive technology.
Re:Why Python 3 Exists by heinrich5991 on Thursday December 17, 2015 @05:03PM (#10754911)
I think something like a lexer doesn't need to know about graphemes, it just needs to know which codepoints are XID_start and XID_continue for identifiers and hardcoded codepoints for everything else.

EDIT: Or a JSON parser, etc.

Re:Why Python 3 Exists by the_mitsuhiko on Thursday December 17, 2015 @05:12PM (#10754967)
A lexer scans LTR. It never jumps codepoints ahead. Indexing in a lexer never makes sense.
Re:Why Python 3 Exists by tomp on Thursday December 17, 2015 @06:46PM (#10755574)
Obviously, it does; instead of allocating a new string for every token, you can just save the start & end index of a token, and actually allocate a new string only in the case where the parser actually requests/uses that token in the AST.
Re:Why Python 3 Exists by nostrademons on Thursday December 17, 2015 @09:16PM (#10756190)
But the type signature of that token should make it clear those are byte offsets and not character offsets. You can use them for byte-wise operations like memcpy, or to explicitly decode them into a string using the original encoding of the string. You cannot use them for string-wise operations like substring without explicitly casting or converting them to a string.

It seems like this whole mess comes from conflating bytes and characters. They are not the same, any more than integers and booleans are the same just because you can jnz in most assembly languages.

Re:Why Python 3 Exists by Veedrac on Friday December 18, 2015 @01:15AM (#10756969)
> You cannot use them for string-wise operations like substring without explicitly casting or converting them to a string.

Why on earth not? Any slice of a string (with a non-pathological internal encoding) with ends on codepoint boundaries is itself a valid string. Losing that information sounds like a whole lot of pointless hassle and potential for harm. It also forces the internal string encoding to be part of the interface, since taking a slice of a string and getting a byte array requires knowing how the string is encoded to use it, but getting another string means that such details are private.

It's true that they're byte offsets, but that doesn't matter. The parser doesn't care whether it's on the 10543th grapheme or the 10597th. It just wants to know where its stuff is.