News for hackers. Stuff that matters.
Re:Swift UTF-8 String by sonnyblarney on Friday March 22, 2019 @12:49AM (#19459704)
If I ever wrote a language, handling strings wrt unicode would be the first thing I would fix so so that it's clear, concise and especially consistent.

It bothers me that in 2019 it can still be difficult to manage even basic things with strings, and yet, whenever we talk/argue/bike-shed about 'languages' we get caught up in intellectual meandering. Please, give us strings that make sense. Then argue about monads.

Re:Swift UTF-8 String by asveikau on Friday March 22, 2019 @01:08AM (#19459788)
I don't think a lot of people realize that Unicode itself, in any representation, is complex, and there is no one abstraction that will make it always easy in all cases. So I think your goal is unobtainable.

Some of that complexity is because world languages themselves are not simple. (eg. If you were to ditch Unicode and start from scratch, it would remain hard to do bi-di text, to pick a random example.)

Re:Swift UTF-8 String by sonnyblarney on Friday March 22, 2019 @01:23AM (#19459841)
Totally see what you are saying (the underlying issue is language itself), but it's pragmatically obtainable.
Re:Swift UTF-8 String by nicky0 on Friday March 22, 2019 @04:27AM (#19460479)
The think is, strings that make sense has been a major goal of Swift, that's why they have been iterating on it so heavily. FWIW I think Swift's is probably one of the cleanest String abstractions out there with respect to unicode correctness balanced against efficiency and usability.
Re:Swift UTF-8 String by asveikau on Friday March 22, 2019 @11:40AM (#19463604)
To be clear I applaud successful attempts to make it better, what I doubt is a possibility to make the problem completely go away without users of the api ever having to think.
Re:Swift UTF-8 String by masklinn on Friday March 22, 2019 @08:20AM (#19461810)
Ah, so your solution is the same old canard: just ignore the complexity and call it pragmatic.
Re:Swift UTF-8 String by sonnyblarney on Friday March 22, 2019 @03:20PM (#19465947)
No. How about a string that returns a Unicode character length?

That's not 'ignoring complexity' and it's pragmatic.

That would help in English as it's really hard today just to deal emojis.

Re:Swift UTF-8 String by asveikau on Friday March 22, 2019 @03:40PM (#19466135)
The point I tried to make: I suspect if you dug really deep into what is a character or glyph you might find "character length" is an imprecise and messy notion, with differing performance costs associated with differing definitions which may or may not apply to you.

So even if you pick one definition and decide to live with it always... You'll still have to know what tradeoffs you (or an API designer or implementer) made.

Re:Swift UTF-8 String by sonnyblarney on Friday March 22, 2019 @03:54PM (#19466259)
First - a combined unicode length is definitely universally useful. Second - a grapheme length is universally useful. Third - I didn't say 'pick one definition' - you did. A simple function which took as a parameter some kind of linguistic construct would be immensely useful in cases of choice/ambiguity.

'It's complicated' is the utterly the wrong answer as to why we 'can't have better strings'.

Strings in almost every programming language were designed before Emjojis and true internationalization. If we designed them today, they would have support for dealing with most of the problems we face.

And FYI - there are not infinite corner cases. They are limited, and we can develop constructs for dealing with all of them.

Re:Swift UTF-8 String by asveikau on Friday March 22, 2019 @05:34PM (#19467199)
There are myriad problems today that 99% of developers couldn't be bothered to think about, but can be a source of bugs. eg. I'm going to guess every time you call tolower() have you haven't thought "I wonder what this does to Turkish I", but it's still there. An API won't fix us from having to think, and having ambiguities occasionally become bugs. (The fact that you've acknowledged the need for multiple string length functions -- I agree with it, but it also means you'll sometimes use the wrong one.)

As I said in another comment, I still applaud efforts to improve the status quo.

Re:Swift UTF-8 String by masklinn on Friday March 22, 2019 @04:16PM (#19466464)
> No. How about a string that returns a Unicode character length?

That doesn't mean anything. Do you mean a code unit, a codepoint, a grapheme cluster, if the latter is it legacy, extended or tailored, how do you handle the locale dependency, … All of them can be useful in some context, all of them are length. And if anything, the truly useful one is the first one (because it tells you how much room you need to store something).

> That's not 'ignoring complexity' and it's pragmatic.

It's true that it's not ignoring complexity, because you're ignorant of it to start with. Which isn't really different.

> That would help in English as it's really hard today just to deal emojis.

It… really isn't.

Re:Swift UTF-8 String by sonnyblarney on Friday March 22, 2019 @07:40PM (#19467984)
Totally wrong. Your implied solution is to just have every single developer, everywhere, to be a linguistics expert and to handle the myriad of international cases for basically everything.

This is totally the wrong approach.

Language is complex and sometimes ambiguous, but there are definitely a series of tools that should be integrated into every 'String' that can facilitate most problems.

Not every company is trying to satisfy Hindi speakers.

"That doesn't mean anything" This is false. It absolutely means something, you can't brush it aside by inventing a separate series of related measurements. Should we have those other measurements? - yes, in many cases, we probably should have them as well - which the whole point. We desperately need a series of common tools built into String, some of them which provide more nuance.

"It's true that it's ignoring complexity, because you're ignorant to start with"

Well, since I have a background in NLP, I've developed writing recognition and word prediction algorithms, as well as language models for dozens of languages for products used by millions of people every day ... maybe I'm not as ignorant as your snide remark suggests?

"> That would help in English as it's really hard today just to deal emojis.

It… really isn't."

Uh, yes, it absolutely would help, and it's painfully obvious. In most languages, which cover the exceeding majority of markets for most apps made, simply knowing the character length (i.e. 'visible character' as to avoid confusion over messy nomenclature here) is very useful. In Latin languages, after Unicode normalization and other kinds of cleaning, Emojis present a very common case for confusion.

There are very small set of tools (or rather, a slightly better, more standardized String approach) which would solve most of the problems for most companies developing for most markets. There are even a few extra tools which would solve, pragmatically all of them.

Re:Swift UTF-8 String by sonnyblarney on Friday March 22, 2019 @07:26PM (#19467896)
Totally wrong. Your implied solution is to just have every single developer, everywhere, to be a linguistics expert and to handle the myriad of international cases for basically everything.

This is totally the wrong approach.

Language is shifty, but there are definitely a series of tools that should be integrated into every 'String' that can facilitate most problems.

"That doesn't mean anything" This is false. It absolutely means something, you can't brush it aside by inventing a separate series of specific measurements. Should we have those other measurements - yes, in many cases, we probably should have them as well -> that's the whole point.

Re:Swift UTF-8 String by saagarjha on Friday March 22, 2019 @12:58AM (#19459750)
> give us strings that make sense

This is really, really hard.

Re:Swift UTF-8 String by BigWhale on Friday March 22, 2019 @08:43AM (#19461988)
In what language do you want them to make sense? A Hindi string would be very different to an Arabic strig, or a japanese string, or an emoji string.
Re:Swift UTF-8 String by Twisell on Friday March 22, 2019 @04:46AM (#19460576)
Tell that to the Python community where switch to UTF8 was/is a huge battle!

Swift is meant to be retro compatible with Obj-C and all. As someone pointed out windows in the meantime have basically no announced battle plan to transition out of UTF16.

I this game latecomer were rewarded because they started with UTF8 or switched from ASCI. But innovators actually tried UTF16 in the mean time an now have to do the job twice in a retro-compatible manner.