I mean the reserved words in my lang are in English, and the user-created identifiers must be ASCII at the moment. I could extend the latter to some wider range but my question was more about providing built-in Unicode string handling.
My Buffer class lets you allocate bytes and it's up to your code to interpret those bytes.
I get your concern though and realize parsing
let foo = "<unicode>"
would be a problem.Apologies if I come off a bit strong on this one. I deal mostly with non “American” text so it’s a pain point. Unicode is so messy, trying to push handling it on to each individual programmer is not an ideal situation.
Let’s just say, no one in Greece, Iceland, or Korea is sitting around wondering if their new programming language should support their alphabet natively.
Imagine being from a country where a language’s built in Strcmp doesn’t work?
No problem !
> Imagine being from a country where a language’s built in Strcmp doesn’t work?
Problem is it's seemingly a massive headache. Citing the article, "Unicode handling is always best left to a library".
I could bundle the recommended lib (ICU) into my lang's stdlib but it seems quite big...
Imagine how you would feel about a language that supports Arabic by default and needs special foo to work with American English?
You need to start thinking of characters as a type. Characters do not fit in bytes unless you’re American.
UTF-8 is a reasonable compromise though.