Then you can have a nice, user-friendly string class for basic UTF-8 text, which is pretty easy. Ignore sorting and grapheme clusters (those probably belong in libraries, and they require fairly large tables). Consider providing a library function to iterate over UTF-8 "characters" (as unpacked 32 bit Unicode code points).
This is one of the sweet spots in language design, and it provides enough structure for third-party libraries to work well together, without everyone reinventing their own string type.
For another good alternative to this approach, see Ruby.
Yes.
Current headache: I have spent the last few days writing a parser for a data representation with far too much embedding. The innermost layer is JSON, representing glTF graphics data. That's UTF-8. That's encapsulated in an obsolete format called "Notation", which is sort of a pre-JSON kind of JSON. It's a byte stream, and has byte counted forms such as
s(5)Hello
This, in turn, is encapsulated in UTF-8 XML. I was coding this in Rust, which led to the discovery that the C++ code it is emulating had problems.Rust has the right concept. Strings are UTF-8, strictly enforced, so you can iterate over them. Byte arrays are arrays of 8-bit unsigned integers. Conversions check.
Classic headache: turning SMS texts into ITA-2 for an antique Teletype machine for steampunk demos. Anything outside of the ITA-2 set (uppercase letters, numbers, and some punctuation) was turned into the long text description of that code point. Emoji text were dutifully hammered out as
(SMILING FACE)(RED HEART)
per the spec. Greetings, Professor Falken.
(SMILING FACE)(RED HEART)That’s how I did it: everything is utf8 (symbols and strings), but you can specifically indicate you want a byte array or nibble or word array etc. And they all work the same transparently.
Imagine how you would feel about a language that supports Arabic by default and needs special foo to work with American English?
You need to start thinking of characters as a type. Characters do not fit in bytes unless you’re American.
UTF-8 is a reasonable compromise though.
My Buffer class lets you allocate bytes and it's up to your code to interpret those bytes.
I get your concern though and realize parsing
let foo = "<unicode>"
would be a problem.Let’s just say, no one in Greece, Iceland, or Korea is sitting around wondering if their new programming language should support their alphabet natively.
Imagine being from a country where a language’s built in Strcmp doesn’t work?
> Imagine being from a country where a language’s built in Strcmp doesn’t work?
Problem is it's seemingly a massive headache. Citing the article, "Unicode handling is always best left to a library".
I could bundle the recommended lib (ICU) into my lang's stdlib but it seems quite big...
EDIT: and most languages made before 1990 too
> handle at least some encodings
What else could it be than Utf8 ? > locale-sensitive casing and collating, and grapheme-cluster access (note that this depends on the font!)
You made up my mind haha. It won't be high-level.If I can write "foo".contains('o') or similar in your language, but then you pretend you don't have strings, nobody will believe you and they're quite right to complain when, inevitably, lots of trickier stuff doesn't work because you didn't bother.
WUFFS doesn't have strings. But WUFFS isn't a general purpose language. Fortran has strings. C has strings (they're terrible but they're clearly strings).
By your logic, couldn't I pretend then to have strings insomuch as C when "foo".contains('o') can be transpiled by my lang as
strchr("foo", 'o') != NULL
My stdlib will provide a (byte) Buffer class with basic low-level methods but I feel like iterating through it in fancy ways should be the concern of the user or 3rd-party libraries.
I fail to see this as part of a programming language.
Am I wrong here ?