News for hackers. Stuff that matters.
Re:Unicode is harder than you think by alcover on Tuesday July 25, 2023 @05:40PM (#36869961)
Currently working on a language, I feel dizzy after reading this.

My stdlib will provide a (byte) Buffer class with basic low-level methods but I feel like iterating through it in fancy ways should be the concern of the user or 3rd-party libraries.

I fail to see this as part of a programming language.

Am I wrong here ?

Re:Unicode is harder than you think by ekidd on Tuesday July 25, 2023 @05:54PM (#36870125)
One good approach is to have separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it."

Then you can have a nice, user-friendly string class for basic UTF-8 text, which is pretty easy. Ignore sorting and grapheme clusters (those probably belong in libraries, and they require fairly large tables). Consider providing a library function to iterate over UTF-8 "characters" (as unpacked 32 bit Unicode code points).

This is one of the sweet spots in language design, and it provides enough structure for third-party libraries to work well together, without everyone reinventing their own string type.

For another good alternative to this approach, see Ruby.

Re:Unicode is harder than you think by Animats on Tuesday July 25, 2023 @08:09PM (#36871451)
> separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it."

Yes.

Current headache: I have spent the last few days writing a parser for a data representation with far too much embedding. The innermost layer is JSON, representing glTF graphics data. That's UTF-8. That's encapsulated in an obsolete format called "Notation", which is sort of a pre-JSON kind of JSON. It's a byte stream, and has byte counted forms such as

    s(5)Hello
This, in turn, is encapsulated in UTF-8 XML. I was coding this in Rust, which led to the discovery that the C++ code it is emulating had problems.

Rust has the right concept. Strings are UTF-8, strictly enforced, so you can iterate over them. Byte arrays are arrays of 8-bit unsigned integers. Conversions check.

Classic headache: turning SMS texts into ITA-2 for an antique Teletype machine for steampunk demos. Anything outside of the ITA-2 set (uppercase letters, numbers, and some punctuation) was turned into the long text description of that code point. Emoji text were dutifully hammered out as

   (SMILING FACE)(RED HEART)
per the spec.
Re:Unicode is harder than you think by alcover on Tuesday July 25, 2023 @08:43PM (#36871707)

  Greetings, Professor Falken.
  (SMILING FACE)(RED HEART)
Re:Unicode is harder than you think by anonzzzies on Wednesday July 26, 2023 @05:55AM (#36875178)
> "Strings are always UTF-8. Anything else is a bug. Deal with it."

That’s how I did it: everything is utf8 (symbols and strings), but you can specifically indicate you want a byte array or nibble or word array etc. And they all work the same transparently.

Re:Unicode is harder than you think by zadokshi on Tuesday July 25, 2023 @06:36PM (#36870657)
You’re definitely wrong. You’re designing a language that only works for “Americans” by default.

Imagine how you would feel about a language that supports Arabic by default and needs special foo to work with American English?

You need to start thinking of characters as a type. Characters do not fit in bytes unless you’re American.

UTF-8 is a reasonable compromise though.

Re:Unicode is harder than you think by alcover on Tuesday July 25, 2023 @07:21PM (#36871091)
I mean the reserved words in my lang are in English, and the user-created identifiers must be ASCII at the moment. I could extend the latter to some wider range but my question was more about providing built-in Unicode string handling.

My Buffer class lets you allocate bytes and it's up to your code to interpret those bytes.

I get your concern though and realize parsing

  let foo = "<unicode>"
would be a problem.
Re:Unicode is harder than you think by zadokshi on Tuesday July 25, 2023 @08:08PM (#36871439)
Apologies if I come off a bit strong on this one. I deal mostly with non “American” text so it’s a pain point. Unicode is so messy, trying to push handling it on to each individual programmer is not an ideal situation.

Let’s just say, no one in Greece, Iceland, or Korea is sitting around wondering if their new programming language should support their alphabet natively.

Imagine being from a country where a language’s built in Strcmp doesn’t work?

Re:Unicode is harder than you think by alcover on Tuesday July 25, 2023 @08:33PM (#36871633)
No problem !

> Imagine being from a country where a language’s built in Strcmp doesn’t work?

Problem is it's seemingly a massive headache. Citing the article, "Unicode handling is always best left to a library".

I could bundle the recommended lib (ICU) into my lang's stdlib but it seems quite big...

Re:Unicode is harder than you think by qalmakka on Thursday July 27, 2023 @03:37AM (#36890210)
To be fair, that has never stopped people from using C and C++ all around the world...

EDIT: and most languages made before 1990 too

Re:Unicode is harder than you think by scatters on Tuesday July 25, 2023 @06:59PM (#36870913)
It depends on the level and domain of your language. A low-level language can get by with just byte arrays. A mid-level language should probably handle at least some encodings, and provide codepoint access. A high-level language should handle locale-sensitive casing and collating, and grapheme-cluster access (note that this depends on the font!).
Re:Unicode is harder than you think by alcover on Tuesday July 25, 2023 @07:36PM (#36871212)
It's a dialect of C, so somewhere between low and mid.

  > handle at least some encodings
What else could it be than Utf8 ?

  > locale-sensitive casing and collating, and grapheme-cluster access (note that this depends on the font!)
You made up my mind haha. It won't be high-level.
Re:Unicode is harder than you think by tialaramex on Tuesday July 25, 2023 @07:28PM (#36871144)
People expect Strings. You don't have to provide strings, but you should be aware that people expect them and if you provide something else but don't clearly label it as not strings there's an excellent chance they'll go "Oh, strings" and then they'll complain that it doesn't work the way they expected.

If I can write "foo".contains('o') or similar in your language, but then you pretend you don't have strings, nobody will believe you and they're quite right to complain when, inevitably, lots of trickier stuff doesn't work because you didn't bother.

WUFFS doesn't have strings. But WUFFS isn't a general purpose language. Fortran has strings. C has strings (they're terrible but they're clearly strings).

Re:Unicode is harder than you think by alcover on Tuesday July 25, 2023 @07:53PM (#36871334)
> If I can write "foo".contains('o') [..] but then you pretend you don't have strings, [..] C has strings

By your logic, couldn't I pretend then to have strings insomuch as C when "foo".contains('o') can be transpiled by my lang as

  strchr("foo", 'o') != NULL
Re:Unicode is harder than you think by tialaramex on Tuesday July 25, 2023 @11:39PM (#36872805)
You could definitely have C's terrible strings, yes. But it is 2023 so that seems like aiming very low indeed